NotImplementedException thrown when trying to compile with function pointers

I try to build my project as the samples suggest (The project doesn't use reflection anywhere, unless 'REFLECTION' is defined which it isn't here).
I get this output with absolutely nothing else
     at ILCompiler.MetadataManager.IsReflectionBlocked(TypeDesc type)
     at ILCompiler.MetadataManager.IsReflectionBlocked(Instantiation instantiation)
     at ILCompiler.MetadataManager.IsReflectionBlocked(MethodDesc method)
     at ILCompiler.DependencyAnalysis.ExactMethodInstantiationsNode.IsMethodEligibleForTracking(NodeFactory factory, MethodDesc method)
     at ILCompiler.DependencyAnalysis.ExactMethodInstantiationsNode.GetExactMethodInstantiationDependenciesForMethod(DependencyList& dependencies, NodeFactory factory, MethodDesc method)
     at ILCompiler.MetadataManager.GetDependenciesDueToReflectability(DependencyList& dependencies, NodeFactory factory, MethodDesc method)
     at ILCompiler.DependencyAnalysis.CodeBasedDependencyAlgorithm.AddDependenciesDueToMethodCodePresence(DependencyList& dependencies, NodeFactory factory, MethodDesc method)
     at ILCompiler.DependencyAnalysis.ScannedMethodNode.InitializeDependencies(NodeFactory factory, IEnumerable`1 dependencies)
     at ILCompiler.ILScanner.CompileSingleMethod(ScannedMethodNode methodCodeNodeNeedingCode)
     at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
  --- End of stack trace from previous location where exception was thrown ---
     at System.Threading.ThreadPoolWorkQueue.Dispatch()
  System.NotImplementedException: The method or operation is not implemented.
     at ILCompiler.MetadataManager.IsReflectionBlocked(TypeDesc type)
     at ILCompiler.MetadataManager.IsReflectionBlocked(Instantiation instantiation)
     at ILCompiler.MetadataManager.IsReflectionBlocked(MethodDesc method)
     at ILCompiler.DependencyAnalysis.ExactMethodInstantiationsNode.IsMethodEligibleForTracking(NodeFactory factory, MethodDesc method)
     at ILCompiler.DependencyAnalysis.ExactMethodInstantiationsNode.GetExactMethodInstantiationDependenciesForMethod(DependencyList& dependencies, NodeFactory factory, MethodDesc method)
     at ILCompiler.MetadataManager.GetDependenciesDueToReflectability(DependencyList& dependencies, NodeFactory factory, MethodDesc method)
     at ILCompiler.DependencyAnalysis.CodeBasedDependencyAlgorithm.AddDependenciesDueToMethodCodePresence(DependencyList& dependencies, NodeFactory factory, MethodDesc method)
     at ILCompiler.DependencyAnalysis.ScannedMethodNode.InitializeDependencies(NodeFactory factory, IEnumerable`1 dependencies)
     at ILCompiler.ILScanner.CompileSingleMethod(ScannedMethodNode methodCodeNodeNeedingCode)
     at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
  --- End of stack trace from previous location where exception was thrown ---
     at System.Threading.ThreadPoolWorkQueue.Dispatch()```

Are you using function pointers?
https://github.com/dotnet/corert/blob/343a4ac490d08633e2653ec5ed7c55fd3e7b36d4/src/ILCompiler.Compiler/src/Compiler/MetadataManager.cs#L662-L663

..yeah that will be it. I was told they were supported by someone else, apologies

@MichalStrehovsky, is there an existing issue tracking function pointer support? Seems like it might be worth it given the language support being added and the runtime support being extended (UnmanagedCallableOnly, etc)

That shouldn't be too hard to fix or workaround, this code is responsible for detecting if it's permitted to reflect upon that. CoreRT supports UnmanagedCallersOnly and uses Calli intrinsics in a lot of places.

@john-h-k do you have sample code that triggers this? And how did you compile it 😅 ? I'm using latest VS2019 preview and .NET 5.0 Preview 5 but this isn't recognized but sharplab on roslyn master branch compiles it.

The sample code is just a method that takes a funcptr. I used the preview tooling on roslyn's MyGet feed

It's basically as simple as:
  <PropertyGroup>
    <RestoreSources>
      https://api.nuget.org/v3/index.json;
      https://dotnet.myget.org/F/roslyn/api/v3/index.json;
    </RestoreSources>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Microsoft.Net.Compilers.Toolset" Version="3.7.0-3.20302.9" />
  </ItemGroup>

Noting that 3.7.0-3.20302.9 was the first nightly to include function pointers and is a week old now.
Also noting that Microsoft.Net.Compilers.Toolset isn't recommended for long term use and that VS IntelliSense (and potentially other bits) won't work for the new language features.

I'm using
<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>net5.0</TargetFramework>
    <AllowUnsafeBlocks>true</AllowUnsafeBlocks>
    <LangVersion>preview</LangVersion>
    <RestoreSources>
      https://api.nuget.org/v3/index.json;
      https://dotnet.myget.org/F/roslyn/api/v3/index.json;
      https://dotnetfeed.blob.core.windows.net/dotnet-core/index.json;
    </RestoreSources>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Microsoft.DotNet.ILCompiler" Version="1.0.0-alpha-*" />
    <PackageReference Include="Microsoft.Net.Compilers.Toolset" Version="3.7.0-3.20302.9">
      <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
      <PrivateAssets>all</PrivateAssets>
    </PackageReference>
  </ItemGroup>

</Project>

and
using System;

namespace ConsoleApp92
{
    unsafe class Program
    {
        public static void Log()
        {
            Console.WriteLine("Works!");
        }

        public static void Test(delegate*<void> del)
        {
            del();
        }

        static void Main(string[] args)
        {
            delegate*<void> a1 = &Log; // Log()

            Test(a1);
        }
    }
}

which compiles fine with dotnet publish -c Release -r win-x64 and prints
> .\bin\Release\net5.0\win-x64\native\ConsoleApp92.exe
Works!

Ah, i've got it, Test has to be generic.

The project doesn't use reflection anywhere, unless 'REFLECTION' is defined which it isn't here

This code is responsible for figuring out whether the type is potentially visible from reflection. Types that are not visible from reflection can have more optimizations applied to them.

is there an existing issue tracking function pointer support?

Nope, it will probably just come down to a bug or two. Most of the system is aware of them (the place that is throwing is aware too, I was just lazy to implement the ~10 lines to handle it.
@Suchiman The method needs to be non-virtual generic to trigger the bug.

Is it basically just IsReflectionBlocked for each param and then the return

The fix will be to cast to a FunctionPointerType, go over the signature and check IsBlocked on each of the types in the signature. I'll mark this up for grabs, but if nobody volunteers, I can fix this later.

@MichalStrehovsky oh here's another one https://github.com/dotnet/corert/blob/343a4ac490d08633e2653ec5ed7c55fd3e7b36d4/src/ILCompiler.Compiler/src/Compiler/DependencyAnalysis/NativeLayoutVertexNode.cs#L382-L384

@MichalStrehovsky oh here's another one

That one you can probably hit with:
class P
{
    static Type Gimme<T>() => typeof(delegate*<T>);
    static void Main()
    {
        typeof(P).GetMethod("Gimme").MakeGenericMethod(typeof(object)).Invoke(null, Array.Empty<Type>();
    }
}

The fix for that right now would be just to pretend it's an IntPtr, but that might change depending on how dotnet/runtime#11354 turns out.

I actually did hit it with this code, i still didn't manage to hit the original issue in of this case 😆
using System;

unsafe class Program
{
    static void Log()
    {
        Console.WriteLine("Works!");
    }

    static void Test<T>(delegate*<void> del)
    {
        del();
    }

    static void Main(string[] args)
    {
        delegate*<void> a1 = &Log; // Log()

        Test<string>(a1);
    }
}


--skip-ssl-validation flag doesn't work

uaac target https://<IP_ADDRESS>:8443 --skip-ssl-validation
failed to access https://<IP_ADDRESS>:8443: Invalid SSL Cert for https://<IP_ADDRESS>:8443/login. Use '--skip-ssl-validation' to continue with an insecure target

This also fails in cf-uaa-lib:
require 'uaa'

issuer = CF::UAA::TokenIssuer.new('https://<IP_ADDRESS>:8443', 'admin', nil, skip_ssl_validation: true)
issuer.client_credentials_grant

ruby ~/uaa.rb
Traceback (most recent call last):
	5: from /Users/kaitingc/uaa.rb:4:in `<main>'
	4: from /usr/local/lib/ruby/gems/2.5.0/gems/cf-uaa-lib-3.13.0/lib/uaa/token_issuer.rb:269:in `client_credentials_grant'
	3: from /usr/local/lib/ruby/gems/2.5.0/gems/cf-uaa-lib-3.13.0/lib/uaa/token_issuer.rb:77:in `request_token'
	2: from /usr/local/lib/ruby/gems/2.5.0/gems/cf-uaa-lib-3.13.0/lib/uaa/http.rb:145:in `request'
	1: from /usr/local/lib/ruby/gems/2.5.0/gems/cf-uaa-lib-3.13.0/lib/uaa/http.rb:156:in `net_http_request'
/usr/local/lib/ruby/gems/2.5.0/gems/cf-uaa-lib-3.13.0/lib/uaa/http.rb:176:in `rescue in net_http_request': Invalid SSL Cert for https://<IP_ADDRESS>:8443/oauth/token. Use '--skip-ssl-validation' to continue with an insecure target (CF::UAA::SSLException)

Hello @ktchen14 , thanks for creating this issue. can you please verify that the configured uaa server certificate is configured correctly i.e are you able to provide the output from openssl s_client -connect <IP_ADDRESS>:8443

@DennisDenuto Let me give you the SSL certificate on that host but note that <IP_ADDRESS>:8443 is an Ops Manager.

Have you ever targeted this environment before with a valid cert? Are there entries for this target in the ~/.uaac.yml file saying skip_ssl_validation: false?

Oh never mind this appears to be due to Ops Manager presenting an invalid certificate in certain cases. You can close this issue. Thanks!


Week 3 handout and lecture

This one is quite heavily new material (hence it took longer and it is currently crap). But I felt it was good to get something down and we can go from there.
Pinging issue #3

reorder complete

ready for @bjmorgan review


[Search v1] Create Search widget

Details

this PR implements this part of the doc: https://docs.google.com/document/d/1u6tBxmoKAEc70jvyZHaKykcfyC5iyEaewfgYXUSSbBc/edit#bookmark=kix.ndabi93vuj44
this handles onyx keys + types, data loading logic and empty states logic
it only introduces data-related parts of Search component - the actual list with items will come in a separate PR
unfortunately there is no api yet, so I had to test it with moc data provided in the doc: https://docs.google.com/document/d/1u6tBxmoKAEc70jvyZHaKykcfyC5iyEaewfgYXUSSbBc/edit#bookmark=id.scrttsoo8ugn

Fixed Issues

$ https://github.com/Expensify/App/issues/39879
PROPOSAL:
Tests

to enter new Search page go to Settings > Troubleshoot > New Search Page
by default you should see the "empty" view
since there is no backend yet, in order to see that onyx correctly displays the data you need to copy-paste Onyx.set command (found below) into browser console - 2 element ids should appear
in general I suggest to review code since without data from backend it's hard to fully test how this looks :(

fake data code snippet
// run this code in browser console to insert fake data:
// hash is the hashed param from URL, by default it should be 96673 for string "all"
Onyx.set(`snapshot_${hash}`, {
            search: {
                offset: 0,
                type: 'transaction',
                hasMoreResults: false,
            },
            data: {
                transactions_1234: {
                    receipt: {source: 'http...'},
                    hasEReceipt: false,
                    created: '2024-04-11 00:00:00',
                    amount: 12500,
                    type: 'cash',
                    reportID: '1',
                    transactionThreadReportID: '2',
                    transactionID: '1234',
                },
                transactions_5555: {
                    receipt: {source: 'http...'},
                    hasEReceipt: false,
                    created: '2024-04-11 00:00:00',
                    amount: 12500,
                    type: 'cash', // not present in live data (data outside of snapshot_)
                    reportID: '1',
                    transactionThreadReportID: '2',
                    transactionID: '5555',
                },
            },
        })

Offline tests

QA Steps

[ ] Verify that no errors appear in the JS console

PR Author Checklist

[x] I linked the correct issue in the ### Fixed Issues section above
[x] I wrote clear testing steps that cover the changes made in this PR

[x] I added steps for local testing in the Tests section
[x] I added steps for the expected offline behavior in the Offline steps section
[x] I added steps for Staging and/or Production testing in the QA steps section
[x] I added steps to cover failure scenarios (i.e. verify an input displays the correct error message if the entered data is not correct)
[x] I turned off my network connection and tested it while offline to ensure it matches the expected behavior (i.e. verify the default avatar icon is displayed if app is offline)
[x] I tested this PR with a High Traffic account against the staging or production API to ensure there are no regressions (e.g. long loading states that impact usability).

[x] I included screenshots or videos for tests on all platforms
[x] I ran the tests on all platforms & verified they passed on:

[x] Android: Native
[x] Android: mWeb Chrome
[x] iOS: Native
[x] iOS: mWeb Safari
[x] MacOS: Chrome / Safari
[x] MacOS: Desktop

[x] I verified there are no console errors (if there's a console error not related to the PR, report it or open an issue for it to be fixed)
[x] I followed proper code patterns (see Reviewing the code)

[x] I verified that any callback methods that were added or modified are named for what the method does and never what callback they handle (i.e. toggleReport and not onIconClick)
[x] I verified that the left part of a conditional rendering a React component is a boolean and NOT a string, e.g. myBool && <MyComponent />.
[x] I verified that comments were added to code that is not self explanatory
[x] I verified that any new or modified comments were clear, correct English, and explained "why" the code was doing something instead of only explaining "what" the code was doing.
[x] I verified any copy / text shown in the product is localized by adding it to src/languages/* files and using the translation method

[x] If any non-english text was added/modified, I verified the translation was requested/reviewed in #expensify-open-source and it was approved by an internal Expensify engineer. Link to Slack message:

[x] I verified all numbers, amounts, dates and phone numbers shown in the product are using the localization methods
[x] I verified any copy / text that was added to the app is grammatically correct in English. It adheres to proper capitalization guidelines (note: only the first word of header/labels should be capitalized), and is either coming verbatim from figma or has been approved by marketing (in order to get marketing approval, ask the Bug Zero team member to add the Waiting for copy label to the issue)
[x] I verified proper file naming conventions were followed for any new files or renamed files. All non-platform specific files are named after what they export and are not named "index.js". All platform-specific files are named for the platform the code supports as outlined in the README.
[x] I verified the JSDocs style guidelines (in STYLE.md) were followed

[x] If a new code pattern is added I verified it was agreed to be used by multiple Expensify engineers
[x] I followed the guidelines as stated in the Review Guidelines
[x] I tested other components that can be impacted by my changes (i.e. if the PR modifies a shared library or component like Avatar, I verified the components using Avatar are working as expected)
[x] I verified all code is DRY (the PR doesn't include any logic written more than once, with the exception of tests)
[x] I verified any variables that can be defined as constants (ie. in CONST.js or at the top of the file that uses the constant) are defined as such
[x] I verified that if a function's arguments changed that all usages have also been updated correctly
[x] If any new file was added I verified that:

[x] The file has a description of what it does and/or why is needed at the top of the file if the code is not self explanatory

[x] If a new CSS style is added I verified that:

[x] A similar style doesn't already exist
[x] The style can't be created with an existing StyleUtils function (i.e. StyleUtils.getBackgroundAndBorderStyle(theme.componentBG))

[x] If the PR modifies code that runs when editing or sending messages, I tested and verified there is no unexpected behavior for all supported markdown - URLs, single line code, code blocks, quotes, headings, bold, strikethrough, and italic.
[x] If the PR modifies a generic component, I tested and verified that those changes do not break usages of that component in the rest of the App (i.e. if a shared library or component like Avatar is modified, I verified that Avatar is working as expected in all cases)
[x] If the PR modifies a component related to any of the existing Storybook stories, I tested and verified all stories for that component are still working as expected.
[x] If the PR modifies a component or page that can be accessed by a direct deeplink, I verified that the code functions as expected when the deeplink is used - from a logged in and logged out account.
[x] If the PR modifies the UI (e.g. new buttons, new UI components, changing the padding/spacing/sizing, moving components, etc) or modifies the form input styles:

[x] I verified that all the inputs inside a form are aligned with each other.
[x] I added Design label and/or tagged @Expensify/design so the design team can review the changes.

[x] If a new page is added, I verified it's using the ScrollView component to make it scrollable when more elements are added to the page.
[x] If the main branch was merged into this PR after a review, I tested again and verified the outcome was still expected according to the Test steps.

Screenshots/Videos

Android: Native

Android: mWeb Chrome

iOS: Native

iOS: mWeb Safari

MacOS: Chrome / Safari
empty

with temp data

MacOS: Desktop

Ready to review but I have a TS error which I do not yet understand, I will be fixing it tomorrow

@luacmartins thanks for the comments, I implemented almost all the changes you requested.
There is one thing which I would like to do in a separate PR, and that is the proper implementation of getListItem/getSections. In order to implement them using the mapping object, probably some generic TS types will have to be introduced, because you need to bind them together using an interface. Otherwise if we just return typesMap[type].ListItem this is any/unknown to TS.
However the next issue to do is actually the items list + TransactionLIstItem component, and those will have to be bound via interface, because thats is what SelectionList expects anyway.
See: https://github.com/Expensify/App/blob/main/src/components/SelectionList/index.tsx#L8
So I would prefer to keep this PR as is and fix this in another PR, where I would have to do a common interface anyways. It will be easier to do it then, right now it will require a bit of TS-hacking to achieve this

That makes sense, let's address the proper implementation of those methods in a follow up then!

@dukenv0307 could you please prioritize reimbursing this PR when you're online? Thank you!

Nice! Let's not block on that comment. I can post the screenshots since I have access to the API internally

:hand: This PR was not deployed to staging yet because QA is ongoing. It will be automatically deployed to staging after the next production release.

🚀 Deployed to staging by https://github.com/luacmartins in version: 1.4.67-0 🚀

platform
result

🤖 android 🤖
success ✅

🖥 desktop 🖥
success ✅

🍎 iOS 🍎
success ✅

🕸 web 🕸
success ✅

🚀 Deployed to production by https://github.com/francoisl in version: 1.4.67-7 🚀

platform
result

🤖 android 🤖
failure ❌

🖥 desktop 🖥
success ✅

🍎 iOS 🍎
success ✅

🕸 web 🕸
success ✅

🚀 Deployed to production by https://github.com/francoisl in version: 1.4.67-7 🚀

platform
result

🤖 android 🤖
failure ❌

🖥 desktop 🖥
success ✅

🍎 iOS 🍎
success ✅

🕸 web 🕸
success ✅


Basic questions about droplets and IP addresses

This looks very interesting for me, as the IP of my single DigitalOcean droplet keeps getting blacklisted by a server that my website users need to authenticate themselves on (via a PHP cURL script on my site). Thankfully it keeps getting un-blacklisted some hours later.
Some basic questions:

Could I set this up remotely on a DigitalOcean droplet that serves my website, instead of on my local machine?

Is it required to set up some DigitalOcean droplets in advance of setting this up? Or just have an account, as the prerequisites say?

Would I still need to have 1 DigitalOcean droplet (currently $4/month minimum) per IP address? Or can this find/generate IP addresses some other way?

Would it be possible to configure ProxyForge so that an IP is prevented from being used in excess of X times per Y minutes? For example, to deal with a server that blacklists after more than 10 requests in a 5 minute period. Or is the IP rotation random?

Hi @Nick-Hope! Thanks for showing interest in ProxyForge. I will try to answer your questions

Could I set this up remotely on a DigitalOcean droplet that serves my website, instead of on my local machine?

Yes, the ProxyForge infrastructure is only intended to be hosted on DigitalOcean servers. You cannot set this up on your local system. However, once the script has executed and all of the resources have been provisioned, you can use the IP address on your local system for testing or anywhere else. Check this video

Is it required to set up some DigitalOcean droplets in advance of setting this up? Or just have an account, as the prerequisites say?

You only need to have your DigitalOcean account with updated billing information.

[!TIP]
For your testing, I would recommend you to signup from my referral link. You will get $200 for 60 days as credit.

Would I still need to have 1 DigitalOcean droplet (currently $4/month minimum) per IP address? Or can this find/generate IP addresses some other way?

Number of droplets depends on how many IPs you want to rotate. Let's suppose you want a pool of $10$ IPs, then you need to have $10$ droplets. Based on the size of droplet you have selected, let's consider $n$ USD, then you will be billed $10 * n + 12$ USD. Additional 12 USD will be billed for Load Balancer.

Would it be possible to configure ProxyForge so that an IP is prevented from being used in excess of X times per Y minutes? For example, to deal with a server that blacklists after more than 10 requests in a 5 minute period. Or is the IP rotation random?

The latter one. IP rotation is randomised. More specifically, the load balancer automatically selected the droplet to pass through the request based on Least Connection algorithm.

@tbhaxor Thank you for your quick and thorough response. Very helpful.

I am closing this issue for now. Feel free to re-open it in case you have any query.


Consider relicense

We should consider re-licensing to MIT/Apache2 as large companies would require support contracts anyways and the current (agpl-v3) license would preclude them from running a trial.

I'd need apache2 as my contributions are required to be apache2.


Graphite URLs are not escaped.

I wondered why my Graphite graphs were not showing anything, and after looking at the Graphite logs I noticed the issue:
<IP_ADDRESS> - - [30/Jul/2014:08:53:19 +0200] "GET /render?format=json&target=legendValue(alias(myohst.apache2.u, 'Open'), 'min', 'max', 'avg', 'last')&from=-3600s&until=-0s HTTP/1.1" 200 184 "-" "Facette"

The characters are not escaped when building the URLs in the graphite.go connector — A correct HTTP request would look like this:
<IP_ADDRESS> - - [30/Jul/2014:08:53:19 +0200] "GET /render?format=json&target=legendValue(alias(myohst.apache2.u,%20%27Open%27),%20%27min%27,%20%27max%27,%20%27avg%27,%20%27last%27)&from=-3600s&until=-0s HTTP/1.1" 200 641 "-" "Facette"

I would be happy to directly fix this in the file but I don't know much about Go. :)

The current 0.3.0 .deb package (amd64 wheezy) seems to still have this issue or a regression of it. Seeing \ escaped quotes in my graphite access log instead of % url escaped ones:
[02/Dec/2015:17:03:44 -0800] "GET /render?format=json&target=alias(collectd.graphite.interface-eth0.if_packets.rx, "series0")&target=alias(collectd.graphite.load.load.shortterm, "series1")&target=alias(collectd.graphite.interface-eth0.if_packets.tx, "series2")&from=-3600s&until=-0s HTTP/1.1"
results in NO DATA instead of graphs.
should be
"GET //render?format=json&target=alias(collectd.graphite.interface-eth0.if_packets.rx,%20\%22series0\%22)&target=alias(collectd.graphite.load.load.shortterm,%20\%22series1\%22)&target=alias(collectd.graphite.interface-eth0.if_packets.tx,%20\%22series2\%22)&from=-3600s&until=-0s HTTP/1.1"
(per graphite log sending the URL via firefox, which auto-fixed/escaped things).


[Question] freez loading page

"Hi,
Suddenly, my WhatsApp integration stopped working—it freezes, showing a static image when trying to interact. The sessions were previously working fine, but when I tried adding a new session, it still froze.
I updated, uninstalled, and reinstalled the application or library, but the problem persists. Now, I don't have any active sessions, and I can't establish a new one. Please help me fix this."

Hi!

Logout/Remove session and rescan QR doesn't help, right?
What docker image do you use? waha-plus or waha-plus:chrome? Try using :chrome if using regular one
Could you check docker logs for any errors/warning right after you started the session, it'd be really helpful :pray:
Could you try setting up a proxy for the session? https://waha.devlike.pro/docs/how-to/proxy/ I've seen blocked IP addresses that behaviour in similar way

Another thing to try if the above doesn't help

Update image to 2024.11.7 version
Set WAHA_WEBJS_CACHE_TYPE=none environment variable (it'll disable cache, just in case of any issues with that)

https://waha.devlike.pro/docs/how-to/config/#webjs


Cygwin fixes

Motivation for this change
This was enough to get git building on cygwin.
The changes should all be limited to stdenv.isCygwin.
Things done

[ ] Tested using sandboxing
(nix.useSandbox on NixOS,
or option build-use-sandbox in nix.conf
on non-NixOS)
Built on platform(s)

[ ] NixOS
[ ] macOS
[ ] Linux

[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
[ ] Tested execution of all binary files (usually in ./result/bin/)
[ ] Fits CONTRIBUTING.md.

@zimbatm I just wanted to make sure it good a solid review, but I'm not aware of any problems, so I removed WIP.
Unfortunately, there's an non trivial conflict now (the cygwin kernel was removed), so I'll have to resolve that and test with latest master.

Thanks, ping me when you fixed the rebase conflicts

@Ericson2314 This patch currently adds back the cygwin kernel you recently removed.  The intention was to allow cygwin to inherit 'unix' packages.  Perhaps 'unix' doesn't really make sense as a kernel family...  What do you think?

@corngood Thanks for the @-mention and good work! So I originally had a cygwin kernel too, but then, following LLVM's example took it out. For now, perhaps you can modify the definition of isUnix instead?
Also, I've been meaning to deprecate stdenv.is*, and use {build,host,target}Platform.is* instead, as the latter are more precise. If you could preempt my deprecation by using that, it would be much appreciated.

@Ericson2314 Rewriting isUnix and removing the unix kernel family seems like a good idea.  I'm thinking it's best to define Unix as a whitelist of:
    Darwin       = { kernel = kernels.darwin; };
    Linux        = { kernel = kernels.linux; };
    SunOS        = { kernel = kernels.solaris; };
    FreeBSD      = { kernel = kernels.freebsd; };
    Hurd         = { kernel = kernels.hurd; };
    NetBSD       = { kernel = kernels.netbsd; };
    OpenBSD      = { kernel = kernels.openbsd; };
    Cygwin       = { kernel = kernels.cygwin; abi = abis.cygnus; };

@corngood yeah feel free to remove the whole notion of "kernel familes" if you like. matchAttrs doesn't really provide much notion of "or"---you can either just define isUnix as a function predicate, or maybe make a matchAnyAttrs that uses list for or?

@corngood yeah feel free to remove the whole notion of "kernel familes" if you like. matchAttrs doesn't really provide much notion of "or"---you can either just define isUnix as a function predicate, or maybe make a matchAnyAttrs that uses a list of patterns with the requirement that just one need match?

@Ericson2314 https://github.com/NixOS/nixpkgs/pull/26021/commits/f851fbace46c5f08d0382680135dcd66aaee57e8 is what I came up with for isUnix.  If I made all the patterns lists, then matchAllAttrs might make sense in attrsets.nix, but I left it local for now.
I didn't completely remove kernel families, because there is at least a conceivable use for them, but I can do that if you'd prefer.

@corngood good decisions on both counts. I just left you one tiny nit---hardly worth mentioning. Other than that, there is the notes on stdenv.is vs hostPlatform.is* on https://github.com/NixOS/nixpkgs/pull/26021#issuecomment-307406495 . I suppose it's fine to just to deal with that latter, but it would be useful if anyone wants to cross compile for cygwin.

@Ericson2314 Is https://github.com/NixOS/nixpkgs/pull/26021/commits/974df1f195793b651b54178e60a7e064ea6dd2d9 what you had in mind?  I'm pretty sure those are all correct as hostPlatform.
I can squash them into the original changes if you prefer.

@corngood a quick skim found one that should actually be buildPlatform.is* but the rest I think are good. Thanks for making these changes!

@Ericson2314 good catch.  I made that one change and squashed each package.

 I suppose I could just throw my hands in the error and merge this like other PRs, but I was waiting in hopes the channels would eventually be fixed and then it could be properly tested.
It's really awful that

We allow master to be broken
We allow merging PRs unrelated to fixing it once it is

Restarting travis now that nixos-unstable is caught up; it might make sense to rebase too.

Hmm, this does look like a mass rebuild. You can use nix-env -f . -qa --out-path | sort and diff to track down hash breakages, or just send this PR to staging instead.

While these changes should all still be valid, some of the core packages have been broken again after rebasing on master.  I'll do some more work on it and either fix the non-cygwin hash breakages or retarget to staging.

I tracked down the cause of libraries being loaded incorrectly from /usr/bin/:
https://github.com/NixOS/nix/pull/1417
So I was able to re-enable checks for a few packages.
I'm still working on the hash changes.

@Ericson2314 it looks like a lot of the changes come things like postPatch = optionalString isCygwin "foo" where postPatch is now "" instead of null.
This seems to be a common pattern for other platforms, and the only way I can think of to fix it is by introducing null, or by using // optionalAttrs.  Both of those could cause problems when someone wants to add cases.
Can you think of a better way?  Should I just move this to staging?

@corngood I'd do ${if hostPlatform.isCygwin then "postPatch" else null} = ... and keep targeting master. Then after this is merged you can clean up in staging. That way, you get to use your changes faster :). I am doing the same thing for my https://github.com/NixOS/nixpkgs/pull/26007

@Ericson2314 I can do postPatch = if isCygwin then "foo" else null and get the same hash.  That seems saner to me than the if on the left of the =.

I like the former because

no extraneous foo = nulls in nix-repl
it's very weirdness is a reminder to fix in staging!

But it really doesn't matter; do whatever you prefer :).

I've prepared a series of hacks that can be merged without changing any hashes on non-cygwin platforms.  I can put that in as a separate commit, and then revert it and PR the revert to staging.  Do you think that would be preferable to just merging this into stable?  Keep in mind I'll probably have a stream of similar changes as I rebase and get more packages working.

Oh you're certainly free to to straight for staging. I personally find waiting for staging to be merged annoying, hence the two-step hacks + hash-breaks.

Ok, I'll move it to staging then.  A bunch of stuff broke in the last rebase, so instead of just building git and a few other random packages, I'm now trying to build a 'stage 2?' stdenv using stagesNix and then I'm going to see if I can build nix itself with that.  That seems a little less arbitrary.

Sounds good. Good luck!

@Ericson2314 rebased on staging.
Building stdenv (especially gcc) is a can of worms.  I'll keep plugging away at it in my spare time, but there's no point in holding this up.

@corngood Fair enough. So is this PR just progress then? Or does git still build?

@Ericson2314 git doesn't currently build due to breakage in ruby, via docbook.

@Ericson2314 actually looks like gitMinimal does build, so it's sufficient for fetchgit, which is what I was originally looking for.

Sounds great; I'm going to merge! Sorry this took so long, @corngood, but excellent job!

@Ericson2314, @corngood: this made all darwin packages evaluate as broken, stopping Hydra builds.  Any ideas?  I bisected the error to 4ac1901d5, but I don't understand the changes enough (yet).

Unix should include Darwin, but doesn't.

Shit, sorry about that.  I prepared a PR to fix isUnix: #27089.  I'm just in the process of reproducing it.

Thanks for the quick fixup.  Job counts on Hydra seem OK now.


override os_family when running on nobara

Using ansible.builtin.set_fact  to override ansible_os_family (which does not affect ansible_facts['os_family'], based on testing
This is because the os_family for Nobara is Nobara, despite it being based on Fedora (ansible_os_family = RedHat). This makes it awkward to run different tasks based on os_family, which is used in multiple locations

A workaround for now has been to symlink any RedHat-specific files to an equivalent Nobara file.
See: 3f7dbeab4eca9acc7ccd0ba08390ed21353fc81e.
This means I don't need to have any special treatment of the ansible facts / built in variables to handle Nobara.
This also means Nobara is relatively lightweight, as it is just an extra symlink somewhere.
However, this does mean that anything where I want to include Nobara has the added step of making the symlink.
Places where this is used, as of https://github.com/Namdrib/ansible-desktop/commit/142246561adbf1568f3ce557a1647719135bfebd:

roles/common
roles/desktop
roles/dev
roles/latex


Update gradle-wrapper.jar

I just noticed that gradle-wrapper.jar was also updated when I ran ./gradlew wrapper.

thanks! merging...


Logging

Closes #16
Adds a logger with a custom formatter that enables PM debugging and shortens exceptions when they occur repeatedly (which happens easily with errors that occur in a draw). The logic to do that is removed from the draw function.

mmm, had to manually start Azure again. What up?

LGTM (except for the build status)


RainflowCounterFKM - wrong results

Hi,
I get wrong results when using the stress.rainflow.RainflowCounterFKM class. The following input should result in one closed loop but does not:
import pylife.stress.rainflow as rainflow

m = np.array([0., 100., 0., 100.])
rfc = rainflow.RainflowCounterFKM().process(m)
print(f"from={rfc.loops_from}", f"to={rfc.loops_to}")

Instead rfc.loops_from and rfc.loops_to are empty. Using the class rainflow.RainflowCounterThreePoint the result is the following: from=[100.0] to=[0.0]
The following code leads to the same result as rainflow.RainflowCounterThreePoint
import pylife.stress.rainflow as rainflow

m = np.array([0., 100., 0., 100., 99.])
rfc = rainflow.RainflowCounterFKM().process(m)
print(f"from={rfc.loops_from}", f"to={rfc.loops_to}")

Is this a problem with the _get_new_turns method? The method rainflow.RainflowCounterThreePoint().process appends the last sample point, while  stress.rainflow.RainflowCounterFKM().process does not.
https://github.com/boschresearch/pylife/blob/8b084f12ef3966a21f22fa9cf36bc7b2c714dc10/pylife/stress/rainflow.py#L241

Thanks for your report.
It is known, that the two algorithms behave slightly different regarding the last loop of the signal and the very first sample. Our algorithm would close the last loop in your case. The FKM one leaves it open and close it only when the next actual turn comes along. We left it like that because when you have many loops the difference is not that big.
We could however easily adjust stress.rainflow.RainflowCounterFKM so that it will pass the following tests:
def test_rainflow_FKM_one_closed_loop():
    signal = np.array([0., 100., 0., 100.])
    rfc = RF.RainflowCounterFKM().process(signal)

    np.testing.assert_array_equal(rfc.loops_from, np.array([100.]))
    np.testing.assert_array_equal(rfc.loops_to, np.array([0.]))
    np.testing.assert_array_equal(rfc.residuals(), np.array([]))

def test_rainflow_FKM_two_closed_loops():
    signal = np.array([0., 100., 0., 100., 0.])
    rfc = RF.RainflowCounterFKM().process(signal)

    np.testing.assert_array_equal(rfc.loops_from, np.array([100.]))
    np.testing.assert_array_equal(rfc.loops_to, np.array([0.]))
    np.testing.assert_array_equal(rfc.residuals(), np.array([100.]))

Would that make sense to you?

Thanks for your quick reply. I agree, that the effect of this is neglegdible for most situations.
My issue is that the FKM one does not loop over the last (and the first) signal point because of the method self._get_new_turns(samples). In other words: the core of the counting algorithm never sees the last and the first signal point. It is therefore less a question of counting per se and more a question of filtering before counting.
Consider the following example. The load sequence m is given and looped over for a first and a second time:
m = np.array([100., 0., 200.,0.])
rfc1 = rainflow.RainflowCounterFKM().process(m)
rfc2 = rainflow.RainflowCounterFKM().process(np.concatenate([m, m]))
print(f"from={rfc1.loops_from}", f"to={rfc1.loops_to}", f"res={rfc1._residuals}")
print(f"from={rfc2.loops_from}", f"to={rfc2.loops_to}", f"res={rfc2._residuals}")

The result of this is independent of the first signal point s = 100, as long it is in the range 0 < s < 200. The first closed loop (closes when reaching the value 200 with range= 100., mean = 50.) is missing in both cases.
When extrapolating, this would have a significant influence. As described in the FKM nonlinear, the total number of sustainable repetitions of the load sequence is calculated by solving D * x + d = 1 for x, where d is the damage sum of the first and D the damage sum of the second repetition of the load signal. Using the proposed algorithm, x would be underestimated.

We don't know what was before the first sample. So in your example [100., 0., 200., 0.] we actually can't tell whether we close a loop when we reach the 200.. If the actual load was [0., 100., 0., 200.0., 0.] we would close a loop at 200.0, whereas if the load was [300., 100., 0., 200., 0.] we would not.
The thing is that get_turns() never gives the first sample and the last sample as a turn because we can't tell if the first and the last sample are turns.
You could precondition rainflow.RainflowCounterFKM by seaking in a zero sample at the beginning like
rfc = RF.RainflowCounterFKM().process(np.array([0.])).process(m)

Then the loop 100 → 0 → 100 would be counted as closed.
Question is whether we always should assume such a zero sample in the beginning. This could be done easily by calling .process(np.array([0.]) in rainflow.RainflowCounterFKM.__init__(). Opinions?

I think it would be useful to have two options when initializing rainflow.RainflowCounterFKM.__init__():

No input (default): call .process(np.array([0.]) in rainflow.RainflowCounterFKM.__init__(). This would eliminate the (for me) unexpected behavior regarding the first sample point.

Initialize with a value for residuals and a value for ir (optional). These values could also be the default ones (residuals=[], ir=1) to get the current behavior.

For the last sample point, we can tell if the loop closes even if we cant tell if its a turning point. But taking this into account may cause a large overhead especially if there are many sample chunks.

Closing this issue in favor of #22.


[k8s-keystone-auth] Add e2e test script

The binaries affected:

[ ] openstack-cloud-controller-manager
[ ] cinder-csi-plugin
[x] k8s-keystone-auth
[ ] client-keystone-auth
[ ] octavia-ingress-controller
[ ] manila-csi-plugin
[ ] manila-provisioner
[ ] magnum-auto-healer
[ ] barbican-kms-plugin

What this PR does / why we need it:
Add a test script for k8s-keystone-auth webhook.
Follow-up PRs:

Change the k8s-keystone-auth CI job in openlab repo to call this script for testing.

Which issue this PR fixes:
fixes #954
Special notes for reviewers:
Release note:
NONE

/test cloud-provider-openstack-acceptance-test-keystone-authentication-authorization

Thanks.
/lgtm

Merge now, I need to test with theopenlab during the weekend.
/approve


compiler freezes compiling a recursive data type

my compiler freezes when compiling the following snippet of code. This was reproduced by people in the elm irc channel.
I am not sure that this code should compile (I may need List.map instead of concatMap), but in any case it shouldn't freeze the compiler.
import Dict exposing (Dict)

type Trie comparable = Trie Bool (Dict comparable (Trie comparable))

toList : Trie comparable -> List (List comparable)
toList (Trie final dict) =
    List.concatMap (\(key, trie) -> key :: toList trie) (Dict.toList dict)

system: Linux mint with elm 0.16, installed via npm.
In addition, the elm playground at elm-lang.org/try doesn't compile this snippet either, and displays a "server refused connection". Maybe some feedback can be given? (something like "The server could not compile your code")
A possible cause is the recursive data structure. Using a fold over this data structure to try get a List of elements also freezes the compiler.
UPDATE
when the type signature is removed, this code does compile, but its type signature was incorrect all along. When changing List.concatMap to List.map, the compiler freezes. Other ways of arriving at this type signature also result in a compiler freeze.
toList : Trie comparable -> List (List comparable)
toList (Trie final dict) =
    List.map (\(key, trie) -> key :: toList trie) (Dict.toList dict)

It seems there are still cases where the compiler does not correctly catch infinite types. I have been tracking that #1231, so I'm going to close this issue and add a reference to it over there. This way I know to circle back to this one.


RaisedButton fullWidth property

Problem description
When setting the property: fullWidth={true} the button does not alter width
This only started after upgrading from 0.14 to 15
If I set the style attribute with style={{width: '100%'}} I do get the desired results.
Steps to reproduce
<RaisedButton
    label='Search'
    fullWidth={true}
    secondary={true}
    onTouchTap={this.handleFetchReport.bind(this)}
  />

Versions
0.15.0

Material-UI:
React:
Browser:

@mbrookes seems like a prop we should get rid of

Yeah, that one always bugged me. I suggest we 'wontfix' this issue (no offence @pulpfree 😄 ), as it's removed in Button.

np @mbrookes , knowing that I can make styling adjustments.
Thanks,
ron

We just ran into this issue. Just confirming this is still the case. If so, I can PR the documentation for RaisedButton.


how about mobilenet flops, compared with squeenzenet

I've added MobileNet, will add MobileNet-v2 once some inconsistencies have been resolved.


feat(airflow) Override datajob external_url

The current url link may not be that useful for the two reasons following, which is why this pr is brought up.

taskinstance page always times out for us due to the huge amount of DAGs we have.
taskinstance page is less informative compared to DAG page

Checklist

[x] The PR conforms to DataHub's Contributing Guideline (particularly Commit Message Format)
[ ] Links to related issues (if applicable)
[ ] Tests for the changes have been added/updated (if applicable)
[ ] Docs related to the changes have been added/updated (if applicable). If a new feature has been added a Usage Guide has been added for the same.
[ ] For any breaking change/potential downtime/deprecation/big changes an entry has been made in Updating DataHub

@gp1105739 I'm trying to understand what override_datajob_url might get set to - looks like it's supposed to be a template URL?

@gp1105739 I'm trying to understand what override_datajob_url might get set to - looks like it's supposed to be a template URL?

Sure, it can be something similar to {base_url}/dags/{datajob.flow_urn.get_flow_id()}/grid?task_id={task.task_id}


Help, cannot infer type-variable(s) D,T,V

version: 0.2.2-SNAPSHOT
issue: cannot compile
error stack:
error: method newCall in class ApolloClient cannot be applied to given types;
return checkResponse(apolloClient.newCall(new Unfollow(builder.build())).execute())
^
required: Operation<D,T,V>
found: Unfollow
reason: cannot infer type-variable(s) D,T,V
(argument mismatch; Unfollow cannot be converted to Operation<D,T,V>)
where D,T,V are type-variables:
D extends Data declared in method <D,T,V>newCall(Operation<D,T,V>)
T extends Object declared in method <D,T,V>newCall(Operation<D,T,V>)
V extends Variables declared in method <D,T,V>newCall(Operation<D,T,V>)

Did you invalidate cached gradle dependencies ?

Because we publishing breaking changes to api module under the same snapshot version, you should refresh your gradle cached dependencies --refresh-dependencies


Resolve @mf-types.zip from remote adress if public path is relative

Clear and concise description of the problem
Currently the remote manifest is loaded via the remote url with the exception of the replace localhost function.
The corresponding types are resolved by the public path given in the manifest.
If the public path is relative the resolution of the types fail as it looks locally for the zip file
const publicPath =
        'publicPath' in manifestJson.metaData
          ? manifestJson.metaData.publicPath
          : new Function(manifestJson.metaData.getPublicPath)();

      remoteInfo.zipUrl = new URL(
        path.join(addProtocol(publicPath), manifestJson.metaData.types.zip),
      ).href;

Suggested solution
If the public path is relative use the domain of the remote to resolve the @mf-types.zip
Alternative
Provide an additional field for the remotes to provide a url to resolve the types from
Additional context
We are running multiple smaller apps in a kubernetes/docker compose setup behind a reverse proxy in the same domain. Therefore we use relative paths for assets to ease the use of different stages
Validations

[X] Read the Contributing Guidelines.
[X] Check that there isn't already an issue that request the same feature to avoid creating a duplicate.

@svenbrosi You mean when relative paths are used, the default prefix for remote is used?

@svenbrosi Can you be more specific with your examples?

We use modernjs as our hosts.
via module federation we resolve the provider from a custom domain.
We have the assetPrefix relative to replace the domain via reverseProxies
tools:{  rspack: (_config, { appendPlugins }) => {       appendPlugins([         new ModuleFederationPlugin({           name: 'catalog_consumer',           remotes: {             app1:               'app1_provider@https://my-custom-domain/app1/mf-manifest.json',           },         }),       ]);     },  output: {     assetPrefix: '/app1',  }
in the manifestJson the assetPrefix is written to the publicPath.
"metaData": {         "name": "app1_provider",         "type": "app",         "buildInfo": {             "buildVersion": "local",             "buildName": "app1"         },         "remoteEntry": {             "name": "static/js/app1_provider.js",             "path": "",             "type": "global"         },         "types": {             "path": "",             "name": "",             "zip"<EMAIL_ADDRESS>            "api"<EMAIL_ADDRESS>       },         "globalName": "app1_provider",         "pluginVersion": "0.1.2",         "publicPath": "/app1/"     },
and this causes the host app to try to resolve the mf-types via the relative path and not via the remotes path.
Hope this clarifies the quesiton

Currently, you can only set public path to a specific path


Added a new model Handwritten Mathematical Equation Solver

Pull Request for PyVerse 💡
Requesting to submit a pull request to the PyVerse repository.

Issue Title
Please enter the title of the issue related to your pull request.
Handwritten Mathematical Equation Solver

[x] I have provided the issue title.

Info about the Related Issue
What's the goal of the project?
Trained a model to solve handwritten mathematical equations.

[x] I have described the aim of the project.

Name
Please mention your name.
Stuti Sharma

[x] I have provided my name.

GitHub ID
Please mention your GitHub ID.
https://github.com/Stuti333

[x] I have provided my GitHub ID.

Email ID
Please mention your email ID for further communication.
<EMAIL_ADDRESS>
[x] I have provided my email ID.

Identify Yourself
Mention in which program you are contributing (e.g., WoB, GSSOC, SSOC, SWOC).
GSSOC

[x] I have mentioned my participant role.

Closes
Enter the issue number that will be closed through this PR.
Closes: #283

[x] I have provided the issue number.

Describe the Add-ons or Changes You've Made
Give a clear description of what you have added or modified.
I have added a new model to the folder Deep Learning

[x] I have described my changes.

Type of Change
Select the type of change:

[ ] Bug fix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds functionality)
[ ] Code style update (formatting, local variables)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] This change requires a documentation update

How Has This Been Tested?
Describe how your changes have been tested.
I have created a complete model and tested on sample images

[x] I have described my testing process.

Checklist
Please confirm the following:

[x] My code follows the guidelines of this project.
[x] I have performed a self-review of my own code.
[ ] I have commented my code, particularly wherever it was hard to understand.
[ ] I have made corresponding changes to the documentation.
[x] My changes generate no new warnings.
[x] I have added things that prove my fix is effective or that my feature works.
[ ] Any dependent changes have been merged and published in downstream modules.

Hi @Stuti333 ,
Could you please close this branch and push again, there is some weird error, I am not sure if its just me.
Thank you!

@ruhi47 PR #969


fix excessive memory usage when parsing large file

There is a memory problem when parsing files (> 2000 lines). The inner loop gets called about 65k times and the app doesn't release memory uses more than a GB.

@SimonStefan It looks really great!! Thank you! I missed a case to load a large file. So, there are some problems such as excessive memory usage and performance.

This still really slow compared to version 1.2.0.
parsing a file with about 6000 lines (8 columns) takes about 45 seconds with this version. It takes less than a second with this version
https://github.com/reidmweber/SwiftCSV
which is forked off of 1.2.0
here is the sample file
samplecsv.txt

@reidmweber Thank you for reporting! Though I don't have enough time to improve the performance, I will try to do it.


[Event Request] Codeunit 5702 Dist. Integration GetSpecialOrders OnBeforePurchLineInit

In the GetSpecialOrders function, after the salesline is found but before the PurchLine.INIT we perform additional validation.  We would need this to occur before the base NAV validation.
Requesting event OnBeforePurchLineInit which would take in parameters for SalesLine, PurchHeader.  Neither would need to be passed by var.
Example:
IF SalesLine.FINDSET THEN
REPEAT
OnBeforePurchLineInit(SalesLine,PurchHeader);
IF (SalesLine.Type = SalesLine.Type::Item) AND ItemUnitOfMeasure.GET…….

Thank you!

Sorry, I also need a parameter for Handled, that when TRUE would skip the base NAV validation around ERROR(Text001
Thanks!


Fix: dependencie resolution typescript, tsec, ...

Fixes typescript dependency version clashes on independent resolve strategies like none yarn works also with yarn

If this gets not merged it will get merged with:
Closeed by:

[x] https://github.com/microsoft/vscode/issues/167505

What is the purpose of this PR?

@alexdima at last needed as long till we got a module cache


Added Elliptical Galaxy

I hate the idea of every single galaxy of a certain type using the same texture, but the rest of of the stuff seems cool, so I'll just revert that one unwanted and unnecessary change.


Pop-ups aren't respecting right margins when the content is too long

Steps to Reproduce

Activate pop-ups on a layer
Add a column that has long content
Check out the pop-up

Current Result

Expected result
To be discussed and defined since this is a bit tricky to solve for every scenario. An ellipsis won't help. Whoever takes this: we should discuss it first between Design and Builder to come up with the best solution.

Not happening anymore.


Here is the main.dart file, exactly how it was written in the documentation.

And Here is where i'm getting the GIFs. My API is working just fine, i'm able to get the GIPHY client random ID

But when i'm trying to access this, i'm getting an error, something about a NULL check on a NULL value

And here's the error am getting on my Device 👇

I'm running this application, on my Android Device, and everything else is working fine, i was using another GIPHY package (giphy_picker)
and it was and still working fine, decided to start using this package cuz it seem to be better than the other. So Help as fast as possible what can i do? to Fix this,
My Flutter and Dart Version 👇
Flutter 3.0.4 • channel stable • https://github.com/flutter/flutter.git Framework • revision 85684f9300 (4 weeks ago) • 2022-06-30 13:22:47 -0700 Engine • revision 6ba2af10bb Tools • Dart 2.17.5 • DevTools 2.12.2

I get the same thing on line 213 of giphy_tab_detail.dart every once in a while.  I can't reproduce.  I could put another null check in with a PR, but not sure if that is the way to fix.


Shadowlist is not scrolling

I tried the example code and shadowlist is not scrolling but it is working fine for flatlist and flashlist
React Native Version: 0.74.0

Could you make sure ShadowListContainer occupies fixed height (e.g. full viewport) this could be adjusted in the same way as you do on FlatList via contentContainerStyle.

Additionally you could find a demo showcasing the functionality here. Please send me a reproducible snippet so I can provide you a better assistance. Thanks!

I have used the same code (did copy and paste) from the example directory

Interesting, would you mind installing v0.0.2 and try again please?

Still same issue

Just noticed when i click on "Load more" button, it gives me react error (i have attached the image)

Thanks for the report, let me check.

I will be available after 6 hours to test it again. Take your time to find and fix it :)

Missed that you are running it on Android, which is currently not supported. I expect to have it done within couple weeks. Thanks!


[WIP] TwoDimensional scrolling foundation

From the 2D scrolling proposal: flutter.dev/go/2D-Foundation ✅  updated 4/25
Fixes https://github.com/flutter/flutter/issues/125505
This adds a mostly abstract foundation for 2D scrolling in Flutter.
With these base classes, developers will be able to construct widgets that scroll in both dimensions and can lazily load their children for the best performance. This implementation is meant to be flexible in order to support different kinds of 2D compositions, from tables to scatter plots.
The upcoming TableView, TreeView, etc widgets (coming soon in flutter/packages) are built on top of this foundation.
This is marked WIP because

There are 3 features for follow up in separate PRs, mainly because it would be difficult to review all of these lumped in with this PR

keep alive  [needs bug]
ensure visible [needs bug]
dual scrollbars

https://github.com/flutter/flutter/issues/122348

I have to move tests over from another branch

A simple concrete test implementation is here to begin facilitating review

I want to add sample code

possibly another separate PR, after outstanding 3 above are added

The design doc is out of date, I need to finish updating it for reference.

Pre-launch Checklist

[x] I read the Contributor Guide and followed the process outlined there for submitting PRs.
[x] I read the Tree Hygiene wiki page, which explains my responsibilities.
[x] I read and followed the Flutter Style Guide, including Features we expect every widget to implement.
[x] I signed the CLA.
[ ] I listed at least one issue that this PR fixes in the description above.
[x] I updated/added relevant documentation (doc comments with ///).
[ ] I added new tests to check the change I am making, or this PR is test-exempt.
[ ] All existing and new tests are passing.

If you need help, consider asking for advice on the #hackers-new channel on Discord.

I am having branch issues. I really do not want to close this, but the Github UI is not updating with the code.

Ok I think my branch issues are behind me. I responded to the last review and left a few questions for feedback. I will push tests next now that I have everything fixed.

Overall, this LGTM. It is missing some test coverage, though. :)

Woohoo! Let's ship it. Kidding.
Added the next block of tests, I had a question for you on one of them, I'll put a comment on it.
Today I filed all follow up issues and notated them in TODOs. The render viewport tests are the only ones left to bring over, I will complete that tomorrow.

The render viewport tests are the only ones left to bring over, I will complete that tomorrow.

I don't know why I say this on Mondays when I know full well Tuesdays are meeting days. 🙃

Still need to fill in a few more tests, the changes to the render viewport broke some of the tests I already had, or the APIs became private. Which I expected, which is why they weren't here to begin with. Should finish having them cleaned up tomorrow barring any hot potatoes. 🥔 🔥

This is ready for review, all tests are now accounted for. :)

hello, how can I use this ? is there any example ?
Thanks


Remove biomarkers.name

🗑 Data to Remove
biomarkers.name is a vestigial column which once recorded the primary name of a biomarker. Later—when aliases were added—a separate table, biomarker_names, would contain the primary name along with a flag indicating if it were primary.
Note that the RDF generator does a SELECT for biomarkers.name but otherwise ignores it.
📝 Code Modification?

Biomarker database: no
RDF generator: yes

Batch №2 since there is a code update.
Note: modifications to portal are not allowed; that means this information is in fact used and this table/column should be preserved.
🩺 Testing Methodology

RDF before/after comparison: yes
In-app testing: yes
Other: N/A

💽 SQL
ALTER TABLE `biomarkers` DROP COLUMN `name`;

🧘 Acceptance
Sign off by @mryan4 by closing this issue

No errors found during testing. This issue is closed.


Updates javascripts path in README.md

Working through the install process here at the University of Cincinnati and noticed a pathing problem with javascripts in the README.md

May not be needed with the inclusion of the lines in the generator.


Does Secor rebalacing lead to missing partitions?

My team has been using Secor for Kafka log persistence for a few months so far. Things work mostly okay, however we have noticed some strange phonomenon. To illustrate the situation precisely, let me list out some facts:

We have about 30 Kafka topics that Secor consumes (mostly low-traffic topics with a few very busy topics).
Each topic has anywhere between 5 and 50 partitions.
We have two AWS EC2 instances (c4.xlarge type) running a Secor process with the identical consumer group name.

In such cases, as far as I understand, Secor automatically distributes partitions for each Secor process. Please let me know if I misunderstood this part.

There have been multiple scaling in/out of Secor instances (not Kafka rebalancing) over the course of the past few months.
We found at least one problematic topic that Secor failed to consume from all partitions. We will call this topic topic1 for simplicity.

I'm not entirely sure how exactly this happened, but one day I have noticed that Secor is entirely missing some of the partitions for topic1 as shown in the following snapshot from an ostrich stats page.
secor.modification_age_sec.topic1.[0].16: 234
secor.modification_age_sec.topic1.[10].21: 2873
secor.modification_age_sec.topic1.[11].21: 3152
secor.modification_age_sec.topic1.[12].22: 929
secor.modification_age_sec.topic1.[13].22: 3286
secor.modification_age_sec.topic1.[14].16: 3653
secor.modification_age_sec.topic1.[15].16: 3664
secor.modification_age_sec.topic1.[16].17: 3713
secor.modification_age_sec.topic1.[17].17: 3713
secor.modification_age_sec.topic1.[18].18: 3709
secor.modification_age_sec.topic1.[19].18: 3709
secor.modification_age_sec.topic1.[1].16: 1176
secor.modification_age_sec.topic1.[1].17: 3632
secor.modification_age_sec.topic1.[20].19: 3651
secor.modification_age_sec.topic1.[21].19: 3655
secor.modification_age_sec.topic1.[22].20: 3718
secor.modification_age_sec.topic1.[23].20: 3721
secor.modification_age_sec.topic1.[24].21: 3613
secor.modification_age_sec.topic1.[25].21: 3626
secor.modification_age_sec.topic1.[26].22: 3664
secor.modification_age_sec.topic1.[27].22: 3664
secor.modification_age_sec.topic1.[2].16: 3669
secor.modification_age_sec.topic1.[2].17: 1868
secor.modification_age_sec.topic1.[2].18: 3670
secor.modification_age_sec.topic1.[3].17: 489
secor.modification_age_sec.topic1.[3].19: 3699
secor.modification_age_sec.topic1.[4].18: 1270
secor.modification_age_sec.topic1.[4].20: 3601
secor.modification_age_sec.topic1.[5].18: 142
secor.modification_age_sec.topic1.[5].21: 3637
secor.modification_age_sec.topic1.[6].19: 139
secor.modification_age_sec.topic1.[6].22: 3685
secor.modification_age_sec.topic1.[7].19: 2625
secor.modification_age_sec.topic1.[8].20: 1155
secor.modification_age_sec.topic1.[9].20: 1155

secor.modification_age_sec.topic1.[43].16: 1070
secor.modification_age_sec.topic1.[44].17: 185
secor.modification_age_sec.topic1.[45].18: 1343
secor.modification_age_sec.topic1.[46].19: 3292
secor.modification_age_sec.topic1.[47].20: 2221
secor.modification_age_sec.topic1.[48].21: 1557
secor.modification_age_sec.topic1.[49].22: 1876

Partition 28-42 (a total of 15 partitions) are missing.
I was wondering whether this is something that may happen during Secor rebalancing.

Probably the secor worker who gets assigned those partitions have problems,
turn on more debug level logging categories can find out those problems.
Or on the zookeeper side I think you can also find the partition to worker
mapping.
On Thu, Dec 26, 2019 at 12:59 AM Sumin Byeon<EMAIL_ADDRESS>wrote:

My team has been using Secor for Kafka log persistence for a few months so
far. Things work mostly okay, however we have noticed some strange
phonomenon. To illustrate the situation precisely, let me list out some
facts:

We have about 30 Kafka topics that Secor consumes (mostly
low-traffic topics with a few very busy topics).
Each topic has anywhere between 5 and 50 partitions.
We have two AWS EC2 instances (c4.xlarge type) running a Secor
process with the identical consumer group name.

In such cases, as far as I understand, Secor automatically
distributes partitions for each Secor process. Please let me know if I
misunderstood this part.

There have been multiple scaling in/out of Secor instances (not
Kafka rebalancing) over the course of the past few months.
We found at least one problematic topic that Secor failed to consume
from all partitions. We will call this topic topic1 for simplicity.

I'm not entirely sure how exactly this happened, but one day I have
noticed that Secor is entirely missing some of the partitions for topic1
as shown in the following snapshot from an ostrich stats page.
secor.modification_age_sec.topic1.[0].16: 234
secor.modification_age_sec.topic1.[10].21: 2873
secor.modification_age_sec.topic1.[11].21: 3152
secor.modification_age_sec.topic1.[12].22: 929
secor.modification_age_sec.topic1.[13].22: 3286
secor.modification_age_sec.topic1.[14].16: 3653
secor.modification_age_sec.topic1.[15].16: 3664
secor.modification_age_sec.topic1.[16].17: 3713
secor.modification_age_sec.topic1.[17].17: 3713
secor.modification_age_sec.topic1.[18].18: 3709
secor.modification_age_sec.topic1.[19].18: 3709
secor.modification_age_sec.topic1.[1].16: 1176
secor.modification_age_sec.topic1.[1].17: 3632
secor.modification_age_sec.topic1.[20].19: 3651
secor.modification_age_sec.topic1.[21].19: 3655
secor.modification_age_sec.topic1.[22].20: 3718
secor.modification_age_sec.topic1.[23].20: 3721
secor.modification_age_sec.topic1.[24].21: 3613
secor.modification_age_sec.topic1.[25].21: 3626
secor.modification_age_sec.topic1.[26].22: 3664
secor.modification_age_sec.topic1.[27].22: 3664
secor.modification_age_sec.topic1.[2].16: 3669
secor.modification_age_sec.topic1.[2].17: 1868
secor.modification_age_sec.topic1.[2].18: 3670
secor.modification_age_sec.topic1.[3].17: 489
secor.modification_age_sec.topic1.[3].19: 3699
secor.modification_age_sec.topic1.[4].18: 1270
secor.modification_age_sec.topic1.[4].20: 3601
secor.modification_age_sec.topic1.[5].18: 142
secor.modification_age_sec.topic1.[5].21: 3637
secor.modification_age_sec.topic1.[6].19: 139
secor.modification_age_sec.topic1.[6].22: 3685
secor.modification_age_sec.topic1.[7].19: 2625
secor.modification_age_sec.topic1.[8].20: 1155
secor.modification_age_sec.topic1.[9].20: 1155
secor.modification_age_sec.topic1.[43].16: 1070
secor.modification_age_sec.topic1.[44].17: 185
secor.modification_age_sec.topic1.[45].18: 1343
secor.modification_age_sec.topic1.[46].19: 3292
secor.modification_age_sec.topic1.[47].20: 2221
secor.modification_age_sec.topic1.[48].21: 1557
secor.modification_age_sec.topic1.[49].22: 1876
Partition 28-42 (a total of 15 partitions) are missing.
I was wondering whether this is something that may happen during Secor
rebalancing.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/pinterest/secor/issues/1130?email_source=notifications&email_token=ABYJP74XNEV6WCOBAKFG5XLQ2RXABA5CNFSM4J7JT5Y2YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4ICVW7PA,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABYJP73FEZ5AWH7LVSHFLZLQ2RXABANCNFSM4J7JT5YQ
.

Thanks for your quick response. I tried to re-produce this error so that I could provide more details, but I was not able to do so. I'll look into the logs and the zookeeper and close this issue for now.

we are observing similar behavior, wherein intermittently secor loses ownership on few partitions.
Took a thread dump during the same time, analysis shows the possible deadlock state in that scenario:
few lines:

0x0000000375332de0ObjectPossible deadlock
Held by secor-consumer-awz_s3_ip-172-31-43-12.ap-southeast-1.compute.internal_2491_189_watcher_executorawaiting notification on [ 0x0000000377f1e898 ] , holding [ 0x0000000375332de0 ]2 thread(s) waiting to take lock Thread-153 Thread-153

0x0000000377f1d368ReentrantLock$NonfairSyncPossible deadlock
Held by ConsumerFetcherThread-secor-consumer-awz_s3_ip-172-31-43-12.ap-southeast-1.compute.internal_2491_189-0-212awaiting notification on [ 0x0000000375345078 ] , holding [ 0x0000000377f1d368 ]1 thread(s) waiting for notifcation: secor-consumer-awz_s3_ip-172-31-43-12.ap-southeast-1.compute.internal_2491_189-leader-finder-thread

0x0000000377f1e378ReentrantLock$NonfairSyncPossible deadlock
Held by ConsumerFetcherThread-secor-consumer-awz_s3_ip-172-31-43-12.ap-southeast-1.compute.internal_2491_189-0-212awaiting notification on [ 0x0000000375345078 ] , holding [ 0x0000000377f1e378 ]1 thread(s) waiting for notifcation: secor-consumer-awz_s3_ip-172-31-43-12.ap-southeast-1.compute.internal_2491_189-leader-finder-thread

<<<

In this leader finder thread is indefinitely waiting for resource on consumer fetcher thread


The station stream function no longer works

Do you know if they have moved it somewhere else, have they purposefully removed it or am i just doing something wrong? Also suggestions for any other ways to do this if that does not work anymore?
in browser
https://radio.garden/api/ara/content/listen/NCzBvLcQ/channel.mp3
{"error":"Not found"}
in cmd
radio-garden station stream -i DfyUx0yR
Error: received status "Not Found" (expected "Found")

Hi @zupermiez! :wave:
It doesn't appear that the Radio Garden API has changed -- other stations (channels) I frequent still work.
Perhaps this station was de-listed from the Radio Garden database. Are you able to find and listen to that station via the Radio Garden app or web UI?

Hm, hang on ... NCzBvLcQ and DfyUx0yR both appear to be place identifiers (Chincoteague Island VA and Uelzen, Germany).
You'll need a station identifier for listening. For example, try V6DXPiZz (WCTG-FM 96.5 in Chincoteague Island VA) or MiD7gD0a (Radio ZUSA in Uelzen, Germany).
In the Radio Garden database, each station is associated with a place. You can provide a place identifier to the stations command to retrieve the stations associated with a place.

big thankyou


How can I see the qualitative results of test images?

Thanks for sharing the code, David.
Is there a way where I can I save the test result images.

No, you'll have to implement it using predict_generator to get the predictions. Should be straightforward and similar to the test] function


When will it arrive on Melpa?

the plugin is submitted and waiting for the PR to merge https://github.com/melpa/melpa/pull/4572

@Compro-Prasad  its arrived on melpa !!! find it from here
https://melpa.org/#/prassee-theme

I have installed it.

👍


Possible regression: As of 1.44 cannot open multiple files from "Go to file..."  menu

Description:
As of 1.44, I am not able to open multiple files from "Go to file..." menu (ctrl + p shortcut) when middle mouse clicked on them. I used it when opening similiar/dependent files (ts, html and css for component, etc.) As of 1.44 middle mouse click only opens last file clicked.
Steps to Reproduce:

Have dependented files (.html, .ts and .css) with same name.
Search for that name in "Go to file..." (ctrl+p shortcut).
Middle mouse click on multiple files.

VSCode Version: 1.44
OS Version: Windows_NT x64 10.0.18363

I found that holding alt + middle mouse button does open multiple files in 1.44. If it is not regression please fill free to close it.

Workaround: configure "workbench.editor.enablePreviewFromQuickOpen": false in settings.


Add new biblatex @dataset entry type

... and promote biblatex @software to regular entry type;
adopting changes introduced in biblatex 3.13, 2019-08-17.

I'm a bit confused about the software change -- looks like all you did was move the line up from the section marked "unsupported" to the other section. This doesn't actually affect anything.  If software is now supported, shouldn't it have some type other than Book?

software is now “supported” by biblatex (i.e., no longer just an alias for misc), so I moved it. One the CSL side, nothing has changed: There still is no type that matches even remotely, so we continue to do the same thing Zotero does for its GUI item type “Computer Program”, i.e., map it to CSL book.

Great, thanks


Git 常用操作命令总结 | 种子的信仰

http://liguoyou.gitee.io/your/2020/05/31/summary-of-git/
GuoYou.Li(李国有)的博客, 用于做学习过程中的笔记、总结以及知识分享

测试 close-issues-notification

Notification settings
Subscribed: Receive all notifications from this issue.

Notification settings
Not subscribed
Only receive notifications from this issue when you have participated or have been @mentioned.

测试 close-issues-notification

引用回复

@228754143

测试 close-issues-notification

@ 你了


added getSessionId

:bulb: Motivation and Context

I need your sessionId to help correlate your events with my own server events.

https://github.com/PostHog/posthog-ios/issues/164
:green_heart: How did you test it?
In my app.
:pencil: Checklist

[ X] I reviewed the submitted code.
[ ] I added tests to verify the changes.
[ ] I updated the docs if needed.
[ ] No breaking change or entry added to the changelog.

NOTE: The code I wrote is this:
    @objc public func getSessionId() -> String {
        if !isEnabled() {
            return ""
        }

        return sessionLock.withLock {
            return sessionId ?? ""
        }
    }

which would have been better as this (return optional, use guard), but I did the above to match your other similar functions...
     @objc public func getSessionId() -> String? {
        guard isEnabled() { return nil }
        return sessionLock.withLock {
            return sessionId
        }
    }

Anyway,  this will help me, thanks!

I didn't bump version or edit changelog - I can if you'd like, but I figured that'd be you folks' department. It'd be nice for this to get deployed with new version so spm would pull it down right away...

Also, not quite sure which tests you'd want... brand new one? Shoehorn into "identify sets distinct and anon Ids"? How would it be done?

@colin-persona thanks for the PR.
the guard option is a nice idea, I can refactor all methods later.
getSessionId can return nil since it's not guaranteed that the session feature is enabled.
feel free to add a changelog entry, just put it under Next and follow the others as an example.
You can add a test on PostHogSDKTest since the sessionId is created during SDK init, you just need to guarantee that the getSessionId returns non-nil.

@colin-persona you can run make format (commit and push if needed) and later make lint to guarantee that CI is happy, and of course make test if you add a test.

closed in favor of https://github.com/PostHog/posthog-ios/pull/170


scrollableContent form input/textarea style: {pointer-events: none;}

browser: google chrome v50.0.2661.94 (64-bit) device mode iPhone 6
desc: click on scrollable-content div cause style: {pointer-events: none;} on input or textarea element
  // overthrow.support === 'none'
  if (overthrow.support !== 'native') {
    element.addClass('overthrow');
    overthrow.forget();
    overthrow.set();
  }

why?

i suggest to remove overthrow.js
or set
    input, textarea {
      pointer-events: auto !important;
    }

@maqic Sorry, i totally overlooked this. Seems to be important.
I can't reproduce the issue. I tried here: http://mobileangularui.com/demo/#/forms, that's also inside a  scrollable-content.
Consider that with an iphone6 overthrow should not be triggered at all.


Named routes?

Any plans to add named routes? Hardcoding hrefs feels very permanent. Named routes allow you to completely separate URL structure logic from your views.
I understand that you're trying to keep this lightweight -- however I feel having named routes and a transition function that takes the parameters as args helps keep all URL related logic in the router.
Otherwise, URL changes mean switching out hrefs in the views.
Thoughts?

---
Want to back this issue? **[Post a bounty on it!](https://www.bountysource.com/issues/26537553-named-routes?utm_campaign=plugin&utm_content=tracker%2F18115217&utm_medium=issues&utm_source=github)** We accept bounties via [Bountysource](https://www.bountysource.com/?utm_campaign=plugin&utm_content=tracker%2F18115217&utm_medium=issues&utm_source=github).

@nathanmarks starting from v1.0.0 the router works with routes declared as plain JavaScript objects where you can add any arbitrary fields other than path and action. For example, you can add name field like so:
const routes = [
  { name: 'home', path: '/', action() { ... } },
  { name: 'account', path: '/account', action() { ... } },
];

The use it in your URL builder utility, for example createHref(routes, 'account') or createHref = createHref.bind(undefined, routes); createHref('account');.
Does this cover your use case?

@koistya Should do the trick! Thanks.


Only reset captcha if being used

This needs a guard otherwise it is called every time the user tries to login with an email that already exists

+1 - downgrading to 1.12.4 fixes the error

Sorry for this :(
tnx!


Coupon API - Cannot apply the coupon if the start date is today regardless the time.

Bug report
Steps to reproduce :

Add a new coupon code on the admin side ({domain}/admin/promotions/cart-rules/create)
Change the start date to today date for example (2 Feb 2021 12 pm) as I am writing
Change the end date to more than 1 month away for example (28 Feb 2021)
After input all the mandatory fields for the new coupon code form, click the Save Cart Rule button.
Next, go to the Postman to test the apply button API ({domain}/api/checkout/cart/coupon)
You will receive a message like a coupon code is invalid.

Another scenario,

Change the start date of the coupon to past days like last month for example (15 Jan 2021, 12 pm)
Next, go to the Postman to test the apply button API ({domain}/api/checkout/cart/coupon)
You will receive a message like success apply the coupon code.

I think the issue is related to the start date of the coupon code.
Let me know if you need more info, thanks :)

@kian92,
Thanks for reporting this. I am checking this If any info needed I will ping you.

Thanks @devansh-webkul, it looks good now.


Sparsemax not actually used in COMET-KIWI, XCOMET-XL/XXL

Hi,
I have been playing around with re-implementing some of your models in Marian and while progressing through the code I noticed that you are not actually using sparsemax for Comet-KIWI and Comet-XL/XXL, instead you are falling back to a softmax.
In both cases you forgot to pass the layer_transformation parameter to its base class:
See here for UnifiedMetric
https://github.com/Unbabel/COMET/blob/2bcf66604b30dcde98565854d5f36026c19f580a/comet/models/multitask/unified_metric.py#L106
and here for XCOMETMetric
https://github.com/Unbabel/COMET/blob/2bcf66604b30dcde98565854d5f36026c19f580a/comet/models/multitask/xcomet_metric.py#L54
In both cases the layer_transformation parameter does not appear in the parameter list of the base class below, but the base class has softmax as the default.
In my re-implementation I am reproducing your exact numbers for Comet-KIWI with a softmax, not the sparsemax. While the sparsemax works fine for COMET-22 ref-based.
It's not clear to me if the model was trained with a softmax or sparsemax, but you might either have a train/inference mismatch here or at the very least your models are doing something different than you expected/described.

Follow-up on that... I am also wondering if you realized that Roberta-XL and Roberta-XXL are pre-norm, while the base model you used for Comet-KIWI is post-norm, but you treat them the same during training/inference. The huggingface implementation is collecting the hidden states without normalization for the XL models with the exception of very last hidden state which is normed.
That seems to mean that the hidden states that you use for your layer-mixing have wildly different magnitudes across layers -- the first and the last one (the most important one?) have very small norms, the ones in-between are unnormed. I am wondering if that wouldn't give you a really hard time during training the xComet-XXL models and skew the weighting during layer mixing?

Yeah, I am currently not looking at the word-level predictions yet, stopped at the regressor-implementation.
Regarding the weights above, the fact that they are near-uniform after softmax despite the that the norms over the hidden states are so different is what made me wonder if proper learning happens or rather some form of saturation (always hard to tell with those neural models).
I would have expected the model to strongly push down the weights for the models with high norms. On the other hand, if this becomes bascially an unweighted arithmetic average then the two very small vectors pull everything down by a lot considering that averages reward outliers. Who knows...

Its the black magic art of NN 🙂


[Bug] ConsumeQueue building exception caused by host downtime

Before Creating the Bug Report

[X] I found a bug, not just asking a question, which should be created in GitHub Discussions.

[X] I have searched the GitHub Issues and GitHub Discussions  of this repository and believe that this is not a duplicate.

[X] I have confirmed that this bug belongs to the current repository, not other repositories of RocketMQ.

Runtime platform environment
OS: CentOS 7.3
RocketMQ version
4.8.x
JDK Version
openjdk version "1.8.0_202"
Describe the Bug

A downtime occurred on the slave node
When the Broker process is started, the following exception log appears during building consumeQueue index.

[BUG]logic queue order maybe wrong, expectLogicOffset:<PHONE_NUMBER> currentLogicOffset:<PHONE_NUMBER> Topic: hlth-center-data QID: 1 Diff: 920

Delete all consumeQueues and back to normal.

Steps to Reproduce

stop the slave node.
delete the last consumeQueue of a queue.
start the slave node.

What Did You Expect to See?
This log should not be printed.
What Did You See Instead?
[BUG]logic queue order maybe wrong, expectLogicOffset:<PHONE_NUMBER> currentLogicOffset:<PHONE_NUMBER> Topic: hlth-center-data QID: 1 Diff: 920

Additional Context
# Possible reason

A power outage on the host causes the loss of unpersisted consumeQueue records in the pagecache.

# Process

ConsumeQueue files are flushed asynchronously, executed once every 1 second

The index is built and stored in the memory, and the disk is flushed once every second, so there will be some index data (the white blocks) is not written to the disk.

A power outage on the host causes the loss of unpersisted consumeQueue records in the pagecache.

After the Broker is started, the current maximum reputOffset is 1004. The index building starts from 1005, so 1002 and 1003 are skipped.

# Verify

Read the wrong consumeQueue file,

last=287986464, new=287986511, file=/home/mi/bak_cq/hlth-center-data/1/00000000005754000000
MessageExt [brokerName=null, queueId=1, storeSize=1914, queueOffset=287986464, sysFlag=0, bornTimestamp=1694089617516}]
MessageExt [brokerName=null, queueId=1, storeSize=1218, queueOffset=287986511, sysFlag=0, bornTimestamp=1694089620422}]

There is a gap. Theoretically, the QueueOffset of the next message should be 287986465, but in fact it is 287986511, a difference of 46. The converted index file length is exactly 920 bytes, which is consistent with the log.

# Hot to fix
1. Memory cache

Add a cache to maintains the flushed offset of each queeu: map <Topic#QueueId, commitLogOffsetPhy>
When exiting abnormally, enter the CQ recover logic and start scanning the commitLog for recovery based on the smallest persistent location recorded in the map.

2. Scan partly commitLogs

Since cq will be flushed periodically, the most recent commitLog files will be affected.
Find the corresponding CommitLog A according to the error log. Then push forward a certain amount of time (3 minutes), find the first CommitLog that exceeds 3 minutes, and then start building from this commitLog

# Linked issue
#1397

seems clear, is it okay to hand it over to me?

Cool, seems like I found this bug when I read the source code, but I don't go into it. 🙉

Cool, seems like I found this bug when I read the source code, but I don't go into it. 🙉

Why not recover from the minimum cmq physical offset when abnormally exiting🤔?

Why not recover from the minimum cmq physical offset when abnormally exiting🤔?

collect all  consumer queue last message's commit log physical offset, and use the minimum as the reputService's reputFromOffset(now is the maxPhysicalPosInLogicQueue).

Why not recover from the minimum cmq physical offset when abnormally exiting🤔?

collect all consumer queue last message's commit log physical offset, and use the minimum as the reputService's reputFromOffset(now is the maxPhysicalPosInLogicQueue).

@suopovate  Starting to recover directly from the minimum physical address of commitLog isn't different from deleting and rebuilding cq. It takes too long and does not seem to be an elegant way.

Why not recover from the minimum cmq physical offset when abnormally exiting🤔?

collect all consumer queue last message's commit log physical offset, and use the minimum as the reputService's reputFromOffset(now is the maxPhysicalPosInLogicQueue).

@suopovate Emmm, looks like this is the first way I mentioned.
Maybe, I think my way is the simplest, just change the 'reputFromOffset' value, finally the data will be normal, and no data loss.🤓
We needn't not keep the cache of consumer queue flushed offset.

Why not recover from the minimum cmq physical offset when abnormally exiting🤔?

collect all consumer queue last message's commit log physical offset, and use the minimum as the reputService's reputFromOffset(now is the maxPhysicalPosInLogicQueue).

@suopovate Emmm, looks like this is the first way I mentioned.
Maybe, I think my way is the simplest, just change the 'reputFromOffset' value, finally the data will be normal, and no data loss.🤓
We needn't not keep the cache of consumer queue flushed offset.

@suopovate

In fact, this also has some disadvantages. Some queues may only generate a few messages, which means that its endOffset is very small. In this case, we will rebuilding almost all cqs.
By the cache, we can add the dimension of time to more reasonably determine the offset which the broker should begin to recover at.

Why not recover from the minimum cmq physical offset when abnormally exiting🤔?

collect all consumer queue last message's commit log physical offset, and use the minimum as the reputService's reputFromOffset(now is the maxPhysicalPosInLogicQueue).

@suopovate Emmm, looks like this is the first way I mentioned.
Maybe, I think my way is the simplest, just change the 'reputFromOffset' value, finally the data will be normal, and no data loss.🤓
We needn't not keep the cache of consumer queue flushed offset.

@suopovate

In fact, this also has some disadvantages. Some queues may only generate a few messages, which means that its endOffset is very small. In this case, we will rebuilding almost all cqs.
By the cache, we can add the dimension of time to more reasonably determine the offset which the broker should begin to recover at.

Emmm, indeed I left out the question.
I will try to fix it in local, thanks for your suggestion.🫡

Hello,I have a question.
If System is exit abnormal, when restart, rocketMq will call recoverAbnormally() to recover consumerQueue and index data.
So why we need to fix the data? 🧐
That's the code :
@Deprecated
public void recoverAbnormally(long maxPhyOffsetOfConsumeQueue) {
    // recover by the minimum time stamp
    boolean checkCRCOnRecover = this.defaultMessageStore.getMessageStoreConfig().isCheckCRCOnRecover();
    boolean checkDupInfo = this.defaultMessageStore.getMessageStoreConfig().isDuplicationEnable();
    final List<MappedFile> mappedFiles = this.mappedFileQueue.getMappedFiles();
    if (!mappedFiles.isEmpty()) {
        // Looking beginning to recover from which file
        int index = mappedFiles.size() - 1;
        MappedFile mappedFile = null;
        for (; index >= 0; index--) {
            mappedFile = mappedFiles.get(index);
            if (this.isMappedFileMatchedRecover(mappedFile)) {
                log.info("recover from this mapped file " + mappedFile.getFileName());
                break;
            }
        }

        if (index < 0) {
            index = 0;
            mappedFile = mappedFiles.get(index);
        }

        ByteBuffer byteBuffer = mappedFile.sliceByteBuffer();
        long processOffset = mappedFile.getFileFromOffset();
        long mappedFileOffset = 0;
        long lastValidMsgPhyOffset = processOffset;
        long lastConfirmValidMsgPhyOffset = processOffset;
        // abnormal recover require dispatching
        boolean doDispatch = true;
        while (true) {
            DispatchRequest dispatchRequest = this.checkMessageAndReturnSize(byteBuffer, checkCRCOnRecover, checkDupInfo);
            int size = dispatchRequest.getMsgSize();

            if (dispatchRequest.isSuccess()) {
                // Normal data
                if (size > 0) {
                    lastValidMsgPhyOffset = processOffset + mappedFileOffset;
                    mappedFileOffset += size;

                    if (this.defaultMessageStore.getMessageStoreConfig().isDuplicationEnable() || this.defaultMessageStore.getBrokerConfig().isEnableControllerMode()) {
                        if (dispatchRequest.getCommitLogOffset() + size <= this.defaultMessageStore.getCommitLog().getConfirmOffset()) {
                            this.getMessageStore().onCommitLogDispatch(dispatchRequest, doDispatch, mappedFile, true, false);
                            lastConfirmValidMsgPhyOffset = dispatchRequest.getCommitLogOffset() + size;
                        }
                    } else {
                        // it is the place that rebuild consumerQueue and index data...🤔
                        this.getMessageStore().onCommitLogDispatch(dispatchRequest, doDispatch, mappedFile, true, false);
                    }
                }
                // Come the end of the file, switch to the next file
                // Since the return 0 representatives met last hole, this can
                // not be included in truncate offset
                else if (size == 0) {
                    this.getMessageStore().onCommitLogDispatch(dispatchRequest, doDispatch, mappedFile, true, true);
                    index++;
                    if (index >= mappedFiles.size()) {
                        // The current branch under normal circumstances should
                        // not happen
                        log.info("recover physics file over, last mapped file " + mappedFile.getFileName());
                        break;
                    } else {
                        mappedFile = mappedFiles.get(index);
                        byteBuffer = mappedFile.sliceByteBuffer();
                        processOffset = mappedFile.getFileFromOffset();
                        mappedFileOffset = 0;
                        log.info("recover next physics file, " + mappedFile.getFileName());
                    }
                }
            } else {

                if (size > 0) {
                    log.warn("found a half message at {}, it will be truncated.", processOffset + mappedFileOffset);
                }

                log.info("recover physics file end, " + mappedFile.getFileName() + " pos=" + byteBuffer.position());
                break;
            }
        }

        processOffset += mappedFileOffset;
        if (this.defaultMessageStore.getBrokerConfig().isEnableControllerMode()) {
            if (this.defaultMessageStore.getConfirmOffset() < this.defaultMessageStore.getMinPhyOffset()) {
                log.error("confirmOffset {} is less than minPhyOffset {}, correct confirmOffset to minPhyOffset", this.defaultMessageStore.getConfirmOffset(), this.defaultMessageStore.getMinPhyOffset());
                this.defaultMessageStore.setConfirmOffset(this.defaultMessageStore.getMinPhyOffset());
            } else if (this.defaultMessageStore.getConfirmOffset() > lastConfirmValidMsgPhyOffset) {
                log.error("confirmOffset {} is larger than lastConfirmValidMsgPhyOffset {}, correct confirmOffset to lastConfirmValidMsgPhyOffset", this.defaultMessageStore.getConfirmOffset(), lastConfirmValidMsgPhyOffset);
                this.defaultMessageStore.setConfirmOffset(lastConfirmValidMsgPhyOffset);
            }
        } else {
            this.setConfirmOffset(lastValidMsgPhyOffset);
        }
        this.mappedFileQueue.setFlushedWhere(processOffset);
        this.mappedFileQueue.setCommittedWhere(processOffset);
        this.mappedFileQueue.truncateDirtyFiles(processOffset);

        // Clear ConsumeQueue redundant data
        if (maxPhyOffsetOfConsumeQueue >= processOffset) {
            log.warn("maxPhyOffsetOfConsumeQueue({}) >= processOffset({}), truncate dirty logic files", maxPhyOffsetOfConsumeQueue, processOffset);
            this.defaultMessageStore.truncateDirtyLogicFiles(processOffset);
        }
    }
    // Commitlog case files are deleted
    else {
        log.warn("The commitlog files are deleted, and delete the consume queue files");
        this.mappedFileQueue.setFlushedWhere(0);
        this.mappedFileQueue.setCommittedWhere(0);
        this.defaultMessageStore.destroyLogics();
    }
}

Hello,I have a question. If System is exit abnormal, when restart, rocketMq will call recoverAbnormally() to recover consumerQueue and index data. So why we need to fix the data? 🧐 Maybe I need to try reproduction this bug. That's the code :

It seems that this case has indeed been considered, but we are using DLedger mode, and the recovery logic of dLedgerCommitLog does not have this process. This should be the cause.

Hello,I have a question. If System is exit abnormal, when restart, rocketMq will call recoverAbnormally() to recover consumerQueue and index data. So why we need to fix the data? 🧐 Maybe I need to try reproduction this bug. That's the code :

It seems that this case has indeed been considered, but we are using DLedger mode, and the recovery logic of dLedgerCommitLog does not have this process. This should be the cause.

That should be it, I made some time to research it 🥲... Thanks, I will be to learn about DLedger.


gov.kg is not allowed according to Registry policy

With public_suffix 1.5.3, gov.kg is not recognized as a valid domain, even though it's listed as a TLD in the Public Suffix List and has a website.

@postmodern this is expected. Here's the .KG definition:
// kg : http://www.domain.kg/dmn_n.html
kg
org.kg
net.kg
com.kg
edu.kg
gov.kg
mil.kg

This definition can be read as:

allow any (.*).kg
(.*).(org|net|...|gov|mil).kg
disallow org.kg, net.kg, ..., gov.kg

Therefore, gov.kg itself is not considered a valid domain as gov.kg is itself a suffix. The longer rule takes precedence, hence gov.kg takes precedence over kg.
➜  publicsuffix-ruby git:(master) rake console
irb -rubygems -I lib -r public_suffix.rb
2.3.0 :001 > PublicSuffix.valid?("foo.kg")
 => true
2.3.0 :002 > PublicSuffix.valid?("gov.kg")
 => false
2.3.0 :003 > PublicSuffix.("foo.kg")
2.3.0 :003 > PublicSuffix.parse("foo.kg")
 => #<PublicSuffix::Domain:0x007fe4cb0d6b20 @tld="kg", @sld="foo", @trd=nil>
2.3.0 :004 > PublicSuffix.parse("foo.gov.kg")
 => #<PublicSuffix::Domain:0x007fe4cb0dd358 @tld="gov.kg", @sld="foo", @trd=nil>

I hope this clarifies your question.


Need to make it easier to expose / extract logging from IActorBenchmarks

Is your feature request related to a problem? Please describe.
I ran into a problem with a misconfigured ShardRegion in https://github.com/Aaronontheweb/actor-benchmarker/pull/7 which caused my clustered benchmarks not to run at all. That issue would have been very easy to diagnose if I could have found a way to inject the xUnit ITestOutputHelper into our logging infrastructure. This is something that we do regularly inside the Akka.TestKit, but the current design of the ActorSystemConfigurator<TActor> doesn't make this particularly easy at the moment.
Describe the solution you'd like
I would like some way to configure the logging both of the IHost and the ActorSystem itself. This will probably require two separate optional configuration methods on the API.
Describe alternatives you've considered
In #7 I tried to implement this on the IBenchmarkConfig interface via a special startup method that attached to the IHostBuilder but that wasn't super workable - I suppose I could have made it workable if I instantiated my configuration inside my test, so maybe this approach is worth revisiting.
Additional context
Probably need to make this work both for unit tests and Benchmark.NET execution.

Goal of this feature is to make it easier for debugging - ideally we want these log settings off during a real benchmark.


Remove branch-name-style in favor of branch rulesets

With GitHub rulesets, this action is no longer required. @SMoraisAnsys pointed out this.

Yeah. While I think that having a convention for branch naming is great, I also think that we shouldn't end up in a situation where after doing my changes and opening a PR, I have to create a new branch and a new PR because the current branch was not correct. It would have been better to not be able to push to the repo in the first place so that I could know that I should rename my branch.


shell/dev files - implement, remove, or move all to wiki or docs

https://github.com/Ladder99/ladder99/tree/main/shell/dev tree has been moved to internal docs.
Remove entire https://github.com/Ladder99/ladder99/tree/main/shell/dev and push back.
No other action is necessary.