Verify contracts

The Problem

Can't verify contracts from hardhat task

The Solution

Add the right modules
Update the router contract to attempt to auto-verify when a router is created

Checklist

[ ] Test manually on test-ui using remote chains.
[ ] Run load tests.
[ ] Update documentation if needed.
[ ] Update CHANGELOG.md.

Hardhat deploy has a built-in verifier. We only can't verify router contracts through it because they are not part of our deployments. Have you verified that the normal hardhat etherscan-verify through the deploy script works? I don't want to break that functionality.

I wasn't able to get it to work as a one-off verify some contract (i.e. verify the FulfillInterpreter or a deployed Router). If I'm doing it wrong it would be great if you can try that in case I'm not doing it correctly

Yeah, there doesn't seem to be a way to one-off verify it. However, there is a flatten tool and it makes it pretty easy to verify one-off contracts.

Ill close in favor of that then

So I actually realized we can totally get this in, because the tasks are named differently. This will actually help things.


Unable to clone repo with ssh git submodule

Running version v3.3.1 with 1d373c4  (which resolved #7469) cherry-picked into the executor
Workflows using git submodules fail to run. The init containers executor fails with ((I've redacted the org and repo names here)...
time="2022-04-12T07:30:28.713Z" level=info msg="Start loading input artifacts..."
time="2022-04-12T07:30:28.713Z" level=info msg="Downloading artifact: code"
time="2022-04-12T07:30:34.414Z" level=error msg="executor error: artifact code failed to load: pkt-line 1: invalid hash text: encoding/hex: invalid byte: U+0052 'R' (ERR \n  myorg/myrepo.git/myorg/mysubmodrepo is not a valid repository name\n  Visit https://support.github.com/ for help)"
time="2022-04-12T07:30:34.415Z" level=info msg="Alloc=216057 TotalAlloc=1090599 Sys=271515 NumGC=21 Goroutines=3"
time="2022-04-12T07:30:34.415Z" level=fatal msg="artifact code failed to load: pkt-line 1: invalid hash text: encoding/hex: invalid byte: U+0052 'R' (ERR \n  myorg/myrepo.git/myorg/mysubmodrepo is not a valid repository name\n  Visit https://support.github.com/ for help)"

The .gitmodules for the impacted repositories are all of the form...
[submodule "dirname"]
        path = dirname
        url = git@github.com:myorg/myrepo.git
        branch = master

I think I've found the problem and will raise an issue in the go-git fork repo.

https://github.com/argoproj-labs/go-git/pull/3


Can I use this module on a cloud storage service like Azure?

First of all, thanks for the brilliant effort. I was wondering how can I set the PATH on a cloud setup such as Azure.  Thanks for your answer in advance

I'm not entirely sure what you mean by PATH. I think you mean environment variables, and if so, then yes should be the answer to your question, however it might be a provider specific way to set those variables so you might want to consult google on how to do that (here's the first one http://stackoverflow.com/questions/34608769/how-and-where-to-define-an-environment-variable-on-azure).


chore(deps): bump https://github.com/jenkins-x/jx-cli to 3.0.78

Update jenkins-x/jx-cli to 3.0.78
Command run was ./promote/main

/retest


Convert implicitly unwrapped optionals to proper optionals

fixed #1320

In my original plan, I wanted to insert optional chaining for any code affected by the refactoring in the same source file, do you think it's a worthwhile and feasible effort?

In my original plan, I wanted to insert optional chaining for any code affected by the refactoring in the same source file, do you think it's a worthwhile and feasible effort?

The question is: What exactly should we do here? I can think of essentially three options:

If we want to make the code compile after the refactoring, we should introduce force unwraps at every location that the refactored because otherwise types might become optional that weren’t before.
We could introduce optional chaining in all member accesses, which possibly makes some code compile but introduces compilation errors where this produces nil where nil wasn’t acceptable before
We don’t do anything at the references and leave it to the user to fix things up

I would be leaning towards option (1) even though I don’t like introducing force unwraps to the user’s code base.

Is it worthwhile?

I think the biggest win with such an implementation is to have a blueprint of a refactoring that uses information from sourcekitd and the index to then perform syntactic refactorings. This refactoring in itself isn’t all that interesting, I think but it could clear the pathway for quite a few interesting refactorings. Some that come to my mind would be:

Changing the order of function parameters
Changing an initializer to a static method
Changing a function to an operator

Having looked into the spec of LSP for a while, I think we need to add a new field data to CodeAction,
public struct CodeAction: ResponseType, Hashable {
...
  /// A data entry field that is preserved between a `CodeActionRequest` and a `CodeActionResolveRequest`.
  public var data: LSPAny?
...

  public init(
    title: String,
    kind: CodeActionKind? = nil,
    diagnostics: [Diagnostic]? = nil,
    edit: WorkspaceEdit? = nil,
    command: Command? = nil,
    data: LSPAny? = nil
  ) {
    self.title = title
    self.kind = kind
    self.diagnostics = diagnostics
    self.edit = edit
    self.command = command
    self.data = data
  }
}

To play it safe, I think we need to keep empty both command and edit in the CodeAction returned and put all necessary information for CodeActionResolveRequest in data, as the official spec only states,

This is usually used to compute the edit property of a code action to avoid its unnecessary computation during the textDocument/codeAction request.

Our most pessimistic expectation may be any well-behaving and non-assuming sourcekit-LSP client should issue a CodeActionResolveRequest after receiving a CodeAction with nil edit and nil command?

Having looked into the spec of LSP for a while, I think we need to add a new field data to CodeAction,

Oh yes, the isPreferred and disabled fields from the LSP spec are also missing and should be added as well.

To play it safe, I think we need to keep empty both command and edit in the CodeAction returned and put all necessary information for CodeActionResolveRequest in data, as the official spec provides no guarantee that nil edit is a necessary and sufficient condition for issuing a CodeActionResolveRequest,
Our most pessimistic expectation may be any well-behaving and non-assuming sourcekit-LSP client should issue a CodeActionResolveRequest after receiving a CodeAction with nil edit and nil command?

It would be good to double-check how a few editors behave but I assume that you’re right here. We should, however, populate edits and command if the client doesn’t support resolving these properties using codeAction/resolve request.

I think the biggest win with such an implementation is to have a blueprint of a refactoring that uses information from sourcekitd and the index to then perform syntactic refactorings. This refactoring in itself isn’t all that interesting, I think but it could clear the pathway for quite a few interesting refactorings. Some that come to my mind would be:

Changing the order of function parameters
Changing an initializer to a static method
Changing a function to an operator

I can envision the necessity of using index for the resolution of this code action, it then implies if the index wasn't available the resolution couldn't proceed. Should we introduce a mechanism to disable code actions when certain conditions aren't met?

I think it would be fine to offer the code action and then return an error response from codeAction/resolve is something goes wrong.


Question: Default dependency injection vs that-depends

This is an opinion question so feel free to treat it as such.

OC (Inversion of Control) container built on that-depends - my DI framework

Is there a reason you picked that-depends instead of the litestar built in dependency injection? Link to litestar DI
I'm getting started on a new project and want to hear your thoughts about the tradeoffs. Thanks!

@SyntaxRules-avant Hi, I can write some thoughts)
Disadvantages of Litestar built in DI:

If you have some consumers or cronjobs or other separate parts of application, than you can't use LiteStar DI in them. Otherwise, Litestar DI can be enough
Also, it can be simpler to test with DI framework, because build in DI only suited for API tests

Cool, I'll check it out. Thanks for the quick reply!


ci: add race detector job

it will run in parallel with normal unit test execution
relates to #286

Will this run on every PR? I think it's a bit overkill like @Wondertan mentioned on Friday. I agree that it will be distracting as we won't know whether the tests are flakey or the code is racey. Can we run on every release instead?

Will this run on every PR? I think it's a bit overkill like @Wondertan mentioned on Friday. I agree that it will be distracting as we won't know whether the tests are flakey or the code is racey. Can we run on every release instead?

Well, considering that release is not happening that often, I think we can introduce the schedule to be at least once per week (let's say Monday 😅 )
I'll upload the updated conf

I think weekly should be fine. cc @Wondertan

Don't we want the tests to be non-flaky too though?

@renaynay, @liamsi We want and race flag that will help us find such hidden ones. The only thing I am worried about is a distraction to fix those for past code and constantly re-running tests to finally get green. This is true unless we find time to allocate for fixing all the existing flakiness. Then we will be able to easily put --race flag on every test run and fix flakiness early, along with races.
I am totally fine with the second option, as this is a technical debt that we have to fulfill anyway.

So, overall I still left the scheduler of the action to be run every Monday 12:00(24hrs) UTC

Let's return to this question once @liamsi is back. After thinking for some time, I think I was wrong, and having -race being run on each test run is the rightest approach that we should stick to.
cc @renaynay

I still like the idea that we run it once a week. @Wondertan

I understand why you like it and still would like to encourage you to embrace and perceive the always with race flag approach due to its correctness and tech debt "preventness".

@liamsi, pls give your opinion on this one. You kinda inspired me to have --race flag always.

I think it should alsways run. tests should be written in a way that this will be possible.

@Bidon15 can you pls update here the workflow to always run things with -race?

Number 1 IMO @Bidon15

Well to clarify, I'd like for it to be a specifically separate action (so we can see test, race, lint etc). I don't mind if it runs in parallel w test, just that it's viewable as a separate action

Agree. Let's do number 1

So this PR has caught 2 new data races
https://github.com/celestiaorg/celestia-node/issues/339
https://github.com/celestiaorg/celestia-node/issues/340
Also there is a failure with Adjecent test but I haven't locally caught it, yet  😓


[v9RC1] Adding image block hangs site, site becomes totally inaccessible

Adding an image block shows an error (an empty notification popup simply saying "ERROR") and hangs site - site unresponsive and unaccessible. Refreshing pages doesn't help, it just hangs.
Only restart of the web server stops it. And now I have lots of this errors:
"Error rendering summary template on page 1 - Unable to locate file for summary template: thumbnail_title_description"
Looks like there is a fundamental bug with images looking at other reports I've made before.

Fixed by #9666

There is a separate issue with the destination picker in this block that I will add to the project board.


Label selector retrieval tests

Slight refactor of deploy command to make it testable

This change is

Reviewed 3 of 3 files at r1.
Review status: all files reviewed at latest revision, all discussions resolved, some commit checks failed.

Comments from Reviewable

Review status: 2 of 3 files reviewed at latest revision, 1 unresolved discussion.

cmd/deploy.go, line 102 at r1 (raw file):
Previously, jellonek (Piotr Skamruk) wrote…
> missing space

Done.

Comments from Reviewable

Reviewed 1 of 1 files at r2.
Review status: all files reviewed at latest revision, 1 unresolved discussion.

cmd/deploy.go, line 102 at r1 (raw file):
Previously, nebril (Maciej Kwiek) wrote…
> Done.

OK

Comments from Reviewable

Review status: all files reviewed at latest revision, 1 unresolved discussion.

Comments from Reviewable

Review status: all files reviewed at latest revision, 1 unresolved discussion.

Comments from Reviewable


[Feature] Please support RWKV

Please add RWKV to model config settings.
You can try RWKV world 7B from RWKV runner https://github.com/josStorer/RWKV-Runner,
and this project natively support openai API format
FYI:


[block]:

/attempt #488

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

Still working on it... Just opened the PR to let you know

/cancel #488


memory_stats.stats.cache not exposed as metric

Hello,
We've tried to use this tool to export some metrics to our Prometheus. What we noticed immediately was that the Memory Usage data reported by our prometheus was completely different than the data reported by AWS Container Insights. We've investigated the issue and learned that the differences are caused by the way AWS calculates the usage of the memory inside of the container.
As you can see in the link below, AWS calculates the usage as the difference between usage and stats.cache (It was changed in 2016):
https://github.com/aws/amazon-ecs-agent/commit/f2019aee016a1be1e340f06147be24367d6f96a6
Could we add this cache metric to the list of exposed metrics. I'd propose naming it as:
memCacheUsageDesc = prometheus.NewDesc(
		"ecs_memory_cache_usage",
		"Memory cache usage in bytes.",
		labels, nil)

With this users of esc_exporter would be able to decide how they want to present the data. We could do it in the same way AWS does (by substracting cache value), show both on a graf etc.

Closed via https://github.com/prometheus-community/ecs_exporter/pull/26.


Heap buffer overflow on GetByteArrayValue function

Hi,
The GetByteArrayValue  function https://github.com/ihedvall/mdflib/blob/adee6fc499d2c1357fa55d798cdbad92c06317be/mdflib/src/cn4block.cpp#L692
does not check maximum allocation and caused heap buffer overflow.
 There are some issues with the file (crashfile.mf4) but at the same time, the file includes all evil MDF types and conversions. I have checked it with the Vectors MDF Validator and my MDF File Viewer. None of them crash so I need to test your exact code. I assume it is the basic example code. I'm running Windows also so it may take some day(s) to figure out the problem.
Preliminary, the file have some issue in the second DG block with the composite CN block (struct_channel_0..7) and the nested structure CN block. It's the last 2 channels.
The Vector MDF Validator detects 13 errors and 64 warnings in the file.

Sure, yes I had to slightly modify the example code. Here it is:
#include <iostream>
#include <mdf/mdfreader.h>
#include <mdf/ichannelgroup.h>
#include <mdf/idatagroup.h>

using namespace mdf;

int main(int argc, char* argv[]) {
    if (argc < 2) {
        std::cerr << "Usage: " << argv[0] << " <path_to_mdf_file>" << std::endl;
        return 1;
    }

    const char* file_path = argv[1];
    MdfReader reader(file_path);  // Open the specified file

    // Read all blocks but not the raw data and attachments.
    // This reads in the block information into memory.
    reader.ReadEverythingButData();

    const auto* mdf_file = reader.GetFile(); // Get the file interface.
    DataGroupList dg_list;                   // Get all measurements.
    mdf_file->DataGroups(dg_list);

    // In this example, we read in all sample data and fetch all values.
    for (auto* dg4 : dg_list) {
        // Subscribers holds the sample data for a channel.
        // You should normally only subscribe on some channels.
        // We need a list to hold them.
        ChannelObserverList subscriber_list;
        const auto cg_list = dg4->ChannelGroups();
        for (const auto* cg4 : cg_list ) {
            const auto cn_list = cg4->Channels();
            for (const auto* cn4 : cn_list) {
                // Create a subscriber and add it to the temporary list
                auto sub = CreateChannelObserver(*dg4, *cg4, *cn4);
                subscriber_list.push_back(std::move(sub));
            }
        }

        // Now it is time to read in all samples
        reader.ReadData(*dg4); // Read raw data from file
        double channel_value = 0.0; // Channel value (no scaling)
        double eng_value = 0.0; // Engineering value
        for (auto& obs : subscriber_list) {
            for (size_t sample = 0; sample < obs->NofSamples(); ++sample) {
                const auto channel_valid = obs->GetChannelValue(sample, channel_value);
                const auto eng_valid = obs->GetEngValue(sample, eng_value);
                // You should do something with data here
            }
        }

        // Not needed in this example as we delete the subscribers,
        // but it is good practise to remove samples data from memory
        // when it is no longer needed.
        //dg4->ResetSample();
    }
    reader.Close(); // Close the file

    return 0;
}

I might have found the error. The level22 channel have invalid offsets causing overrun when reading. I have added some range test to harden the code so it's now returning non-valid values which is a fair result.
I cannot reproduce your error in my environment but if you filter out the level22 channel subscription from your test code, the error should disappear.
The algebraic (formula) conversion is currently not supported. If you need support for that conversion method, please let me know.
I just need to check if the value range to value conversion is correct. It should work so I need to check this. It's currently returning invalid values.

Thanks, in case if you need to reproduce the AddressSanitizer crash above, you can try compile the sample code above with -fsanitize=address, that should give the same error in the issue description. If, not then it means the those are fixed.

I have made some changes regarding the buffer overrun. Tested on Windows.

I have added a fuzzytest.sln and verified that the heap buffer overflow problem is fixed. The file have an invalid index causing the problem. The fix is to set all values invalid for that channel but OK for the remaining channels.


how to sort the query result

how to sort the query result? Only filter found. we want something like sortBy id desc, etc.

At the moment, ORDER BY is not supported. Unfortunately I don't think I have got the time to anything about it before May. You are welcome to implement something and send a pull request, if you have the time :)
Until then, maybe you could use Swift's sort methods instead? They might be a bit slower, depending on your data/queries, but unless you have huge datasets, they will probably do the trick!

I had fork a branch : https://github.com/swifth/SwifthDB . and add support for order by and carthage support. and removed the multiple SQLiteDatatype definition. and fixed others bugs.  But I have no idea how to merge into master.


Right to left support

Hi, I just added pager tab strip into one of my controllers and my app local is RTL but the scroll view does not work correctly and since tabs are populated from right to left but controllers populated from left to the right.
To test that just change app default language to one of the RTL languages such as Arabic ,Farsi or Hebrew

Hello I changed to español and works perfectly, Can you share the code?

Isn't español  a LTR language? I meant for occasions where view direction changes (mirror) to right to left.

I will check for LTR Language(Arabic) and see if it works

In summary the problem is the tabs works correctly but the scroll view still populate view from left to the right so the rightmost tab show the leftmost view and so on ...

@hrsalehi Do you have time to look at it and find out a solution?

try setting your main ScrollView for PagerTabStrip i.e(container view) view semantic to : Force-Right-To-Left and see if it solves the problem

Setting force Right-to-Left semantic on the containerView (UIScrollView) did not solve the issue 👎 Any suggestion that might fix this issue please?

@santiagofm Do you think anything could be done on this one please?

One solution is to reverse the tabs array before setting it to the controller @musbaalbaki

Thanks @mohammadghk, that is actually what I ended up doing. However I am still looking for a cleaner approach using the native iOS RTL support

since I am using autolayout, the tabs section automatically adjusted in RTL language. Now the problem I am facing is that the view controllers slides in from right side in Arabic which is basically opposite.

hi,
I have the same problem. Is it a solution
Thank you

This is not a proper solution  but after way too many hours trying to solve this problem, this actually solved it.
Add these lines inside viewDidLoad method in PagerTabStripViewController
    if #available(iOS 9.0, *) {
        if UIView.userInterfaceLayoutDirection(for: view.semanticContentAttribute) == .rightToLeft {
           print("arabic")
            containerView.transform = CGAffineTransform(rotationAngle: CGFloat(Double.pi))
            for controller in viewControllers
            {
                controller.view.transform =  CGAffineTransform(rotationAngle: CGFloat(Double.pi));

            }
        }
        else
        {
            print("English")
        }
    } else {
        // Fallback on earlier versions
    }

I used AdnanMajeed06 solution by adding the transform function in ViewDidLoad after super.viewDidLoad and without reversing the tabs array by extending UICollectionViewFlowLayout
like so:
extension UICollectionViewFlowLayout {

	override open var flipsHorizontallyInOppositeLayoutDirection: Bool {
		return true
	}

	override open var developmentLayoutDirection: UIUserInterfaceLayoutDirection {
		return UIUserInterfaceLayoutDirection.rightToLeft
	}
}

The tabs start from right to left and the scrollview direction is working like it supoosed to do in RTL

I ended up with another similar solution, which fixes the unexpected rotation if implemented using @AdnanMajeed06 solution.
My solution:
// Add at the end of `viewDidLoad` of `PagerTabStripViewController`
if UIView.userInterfaceLayoutDirection(for: view.semanticContentAttribute) == .rightToLeft {
    containerView.transform = CGAffineTransform(scaleX: -1, y: 1)
    for controller in viewControllers {
        controller.view.transform = CGAffineTransform(scaleX: -1, y: 1)
    }
}

It works fine with RTL and LTR

Nothing works for me and this bug is still open, any new solution here?

Any solution for this issue

stuck in the same issue ...

for 3 or less than thre tabs follwing code works fine for me ...
 self.buttonBarView.semanticContentAttribute = GeneralMethods.getSelectedLanguage() == .arabic ? .forceRightToLeft : .forceLeftToRight
but for more than three tabs above code not working... so I change the logic... XLPagerTabStrip uses CollectionView for top buttons known as buttonBarView ...
So I created a class with name ArabicCollectionFlow
class ArabicCollectionFlow: UICollectionViewFlowLayout {     override var flipsHorizontallyInOppositeLayoutDirection: Bool {         return GeneralMethods.getSelectedLanguage() == .english ? false : true//AppLanguage.switchToRTL     } }
and assign the class to 'buttonBarView'
   `let collectionLayout = ArabicCollectionFlow()
    collectionLayout.scrollDirection = .horizontal
    self.buttonBarView.collectionViewLayout = collectionLayout`


Adds libri-100 default split files

[ ] Tests added / passed
[ ] Associated doc strings if new function
[ ] Passes flake8

@zhampel @MaAleBarr I have added further functionality that, unfortunately, was added to this pull request. I have added a couple of defenses and a test for one of them (I have run local tests on both), please review

No worries. I'll take a look. Thanks for the notification.

Thanks for adding documentation and including tests.


localStorage persisting across sessions

In the latest version of Walaby (0.17.0), I believe I'm seeing the contents of localStorage persist across Wallaby's phantomjs sessions, despite PR #93 which intended to separate localStorage between phantomjs sessions.
I've created a minimal test case in this repo: https://github.com/stevegrossi/wallaby_test
Steps to Reproduce
I generated a brand new phoenix app with the default route going to PageController.index
On that page I have a bit of JavaScript that writes the contents of localStorage to the page, and then sets 'foo' to 'bar':
<script>
  const container = document.querySelector('#localstorage');
  const localstorageContents = "foo = " + localStorage.getItem('foo');
  container.innerText = localstorageContents;
  localStorage.setItem('foo', 'bar');
</script>

The idea being that if localStorage is empty, we can expect foo = null but if foo is set from a previous test run, we'll see foo = bar.
I set up Wallaby according to the README. My only departure from those instructions was not to add async: true to my tests, because in order to test localStorage persisting across tests I needed them to run serially.
Finally, I have two identical Wallaby feature tests that visit the index page and assert that foo = null was written to the page:
IO.puts "session.id: #{session.id}"

session
|> visit("/")
|> assert_has(css("#localstorage", text: "foo = null"))

I also puts the session ID in order to confirm they're using different phantomjs sessoins. When run individually both tests pass:
steve@wallaby_test mix test test/features/page_one_test.exs
session.id: a1729160-67ae-11e7-8935-75b4a57c6225
.

Finished in 1.4 seconds
1 test, 0 failures

Randomized with seed 480326
steve@wallaby_test mix test test/features/page_two_test.exs
session.id: a58affd0-67ae-11e7-8fec-e7d5317a2964
.

Finished in 1.4 seconds
1 test, 0 failures

Randomized with seed 279952

But when run together with mix test test/features, the second fails, presumably because the page contains foo = bar from the previous session's localStorage.set, and not the expected foo = null:
steve@wallaby_test mix test test/features
session.id: a9911ec0-67ae-11e7-9d90-15727110d3bb
.session.id: a9acbd10-67ae-11e7-9d90-15727110d3bb

  1) test localstorage is empty a the start of the test (WallabyTest.PageOneTest)
     test/features/page_one_test.exs:6
     ** (Wallaby.ExpectationNotMet) Expected to find 1, visible element that matched the css '#localstorage' but 0, visible elements were found.

     stacktrace:
       test/features/page_one_test.exs:11: (test)

Finished in 4.4 seconds
2 tests, 1 failure

Thanks for the great writeup @stevegrossi :+1:. I'll take a look and see if I can reproduce this locally.

Thanks! I should mention I have a perfectly valid workaround in my app: simply logging out after each feature test so that the token I persist in localStorage goes away.
And thanks for your hard work on Wallaby!

We should confirm that this isn't broken in chrome. There are known sanitization issues with phantom but I'm planning on replacing phantom with chrome as the default driver. If chrome does not share this issue then I think we're safe to close it.

It looks like this is fine in chrome: https://github.com/mhanberg/wallaby_test
Tested with elixir 1.7, wallaby 0.22 and chromedriver 2.46.628411

Closing due to issue being related to phantomjs, which is now removed.


Expose Client and Query classes directly

These two classes are the main public interface that users are expected to use in the client, so they should be exposed directly under the module.
This allows us to get rid of the deep syntax of imboclient.client.Client, replacing it with from imboclient import Client or just imboclient.Client. The same will be true for the Query class, allowing users to use the class directly from the module (from imboclient import Query) instead of searching through imboclient.url.imagesquery to find Query.
I'm considering if we should expose Query as ImagesQuery to avoid future clashes with UserQuery or similar. Any opinions?

Seems sensible if you ask me. But I see you already concluded. Are you planning more work on this project going onward? 🥇


Adds controllable randomization to range queries in workloads

Description
Allows the user to randomize the values in range queries in a controllable way. This allows reasonable tests of cache performance, where we can't use the same query repeatedly.
To use this, in your workload's workload.py file, you must register a "standard value source" function in the register() function. This just means a function that provides a dict of range values, with keys "gte", "lte", and optionally "format" (for dates). Usually you would want some randomness in this function, so that different values are produced every time you run it. During the workload, on each iteration, values from an output from this function get put into the range section of a query, and overwrite the default values.
This registration is done on an operation and field level. For example, in nyc_taxis, say you wanted to randomize the field "trip_distance" in the operation "distance_amount_agg", to produce random values between 1 and 15. Your workload.py might look like:
def trip_distance_std_value_source():
    gte = random.randrange(1, 7)
    lte = random.randrange(gte, 15)
    return {"gte":gte, "lte":lte}

def register(registry):
    registry.register_runner("delete-snapshot", delete_snapshot, async_runner=True)
    registry.register_standard_value_source("distance_amount_agg", "trip_distance", trip_distance_std_value_source)

Then, to enable randomization when you run the workload, you supply the flag --randomization-enabled. To control the fraction of queries which may be repeated (for example, to mimic a cache hit), you supply the flag --randomization-rf 0.3, which would mean the "repeat frequency" for this workload is 0.3 = 30%. If --randomization-enabled isn't passed, there is no change from the user's perspective and you don't have to register standard value sources.
This works by adding a new WorkloadProcessor called QueryRandomizerWorkloadProcessor. If randomization is enabled, its on_after_load_workload() function will walk through the default workload schedule (the one specified in test_procedures/default.json). For all search operations, it will see if a standard value source was registered for that operation and field type. Then it gives the operation a new ParamSource.
This ParamSource is a lambda supplied by the processor. When it's run, it finds the range section of the query, and replaces its gte/lte (and format, if present) values with the ones provided by the standard value source for that operation and field.
To allow a controllable fraction of "cache hits", we use the rf parameter from above.
If rf = 0.3, then 30% of the time, we want a repeated value. To achieve this, we generate and save some number N of standard values for each op/field pair at the beginning of the workload, using the same standard value source function. (Default = 5000, this is configurable via --randomization-n). We pick one of these saved values to return from the lambda. (We draw the index of this value from the Zipf distribution on N items, which empirically matches the distribution for online cache requests, and has some other advantages too). As the workload goes on, indices will be drawn multiple times, so the query values will repeat.
The other 70% of the time, we generate a totally new value from the standard value source function, which will not have been seen before (a "cache miss").
Issues Resolved
Resolves https://github.com/opensearch-project/opensearch-benchmark/issues/443
Testing

[x] New functionality includes testing

Added UTs testing various components of the workflow. Also tested manually with several different workloads while watching the parameters, to make sure they were changing correctly and the fraction of ones being drawn randomly was correct. Finally, re-ran a cache-related workload that we had run before with a previous, non-generic way of randomizing queries, and made sure the number of hits, misses, evictions, etc matched what we observed then.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Also tested manually with several different workloads while watching the parameters, to make sure they were changing correctly and the fraction of ones being drawn randomly was correct. Finally, re-ran a cache-related workload that we had run before with a previous, non-generic way of randomizing queries, and made sure the number of hits, misses, evictions, etc matched what we observed then.

Could you include some finer details on how you validated this? Any outputs pasted in this PR or the associated issue would be helpful!

Also, could you elaborate on what you mean by "standard value source"? Is this essentially just a customized value?

Also, could you elaborate on what you mean by "standard value source"? Is this essentially a customized value for a specific operation?

Pretty much. It's a function you call to generate a random pair of values for a certain field. In the actual workload, we generate 5000 of these at the beginning and save them, to use for repeated values (mimicking cache hits). To mimic a totally new query that would cause a cache miss, we run the source function again and use that new output. Assuming your standard value source function has a lot of possible outputs, it probably won't have been seen before.
I've updated the PR description to give more details.

Also tested manually with several different workloads while watching the parameters, to make sure they were changing correctly and the fraction of ones being drawn randomly was correct. Finally, re-ran a cache-related workload that we had run before with a previous, non-generic way of randomizing queries, and made sure the number of hits, misses, evictions, etc matched what we observed then.

Could you include some finer details on how you validated this? Any outputs pasted in this PR or the associated issue would be helpful!

Just added some printouts and descriptions to the original PR comment. Let me know if you want more info!

Also tested manually with several different workloads while watching the parameters, to make sure they were changing correctly and the fraction of ones being drawn randomly was correct. Finally, re-ran a cache-related workload that we had run before with a previous, non-generic way of randomizing queries, and made sure the number of hits, misses, evictions, etc matched what we observed then.

Could you include some finer details on how you validated this? Any outputs pasted in this PR or the associated issue would be helpful!

Just added some printouts and descriptions to the original PR comment. Let me know if you want more info!

This is great stuff! Thanks for the concise and supporting information added to the PR description

Left a single comment on a fix regarding the integration tests.


review and update pachctl docs

We should do a once-over on all the help messages for pachctl docs. We definitely should expand and add details as well as unify the language and terms in the help docs.
--split-file for instance could be much clearer defining the delimiter and target-datum-count

Split file updated here: http://pachyderm.readthedocs.io/en/latest/cookbook/splitting.html.  Also, pachctl docs were scrubbed at the 1.5 release.  Closing for now in favor of more specific doc issues.


Android - Elevation style not working on elements rendered inside MaskedView

Bug
I'm trying to render a scroll view inside a MaskedView with LinearGradient as a maskElement.. On Android all elements with elevation style lose their shadow..
Environment info
 System:
    OS: macOS 10.14.5
    CPU: (8) x64 Intel(R) Core(TM) i5-8259U CPU @ 2.30GHz
    Memory: 426.71 MB / 8.00 GB
    Shell: 3.2.57 - /bin/bash
  Binaries:
    Node: 12.3.1 - ~/.nvm/versions/node/v12.3.1/bin/node
    Yarn: 1.16.0 - /usr/local/bin/yarn
    npm: 6.9.0 - ~/.nvm/versions/node/v12.3.1/bin/npm
    Watchman: 4.7.0 - /usr/local/bin/watchman
  SDKs:
    iOS SDK:
      Platforms: iOS 12.2, macOS 10.14, tvOS 12.2, watchOS 5.2
    Android SDK:
      API Levels: 23, 26, 27, 28, 29
      Build Tools: 27.0.3, 28.0.0, 28.0.1, 28.0.2, 28.0.3, 29.0.0, 29.0.0
      System Images: android-27 | Google Play Intel x86 Atom
  IDEs:
    Android Studio: 3.4 AI-183.6<IP_ADDRESS>22156
    Xcode: 10.2.1/10E1001 - /usr/bin/xcodebuild
  npmPackages:
    @react-native-community/cli: ^2.6.2 => 2.6.2
    react: 16.8.6 => 16.8.6
    react-native: 0.60.3 => 0.60.3
  npmGlobalPackages:
    react-native-create-library: 3.1.2

Library version: 0.1.1
Steps To Reproduce

Use MaskedView with LinearGradient element as mask and render inside a view with elevation.

Describe what you expected to happen:

The view inside MaskedView will have a shadow due to the elevation style.

Reproducible sample code
<MaskedView
      maskElement={(
        <LinearGradient
          style={{ height: 200, width: 200 }}
          colors={['transparent', 'white']}
          locations={[0.01, 0.06]}
        />
      )}
>
    <View style={{ width: 100, height: 100, elevation: 5, backgroundColor: 'red'  }} />
</MaskedView>

I did some research and found that replacing LAYER_TYPE_SOFTWARE with LAYER_TYPE_HARDWARE in RNCMaskedView.java line 22 fixes the issue..
Is it a good solution in your opinion?

Tested on react-native 0.61.3 and react-native-masked-view 0.1.2 and it work as expected

I'm experiencing the same issue here :(

Any update? Me too experiencing the same issue 😪

For me changing LAYER_TYPE_SOFTWARE to LAYER_TYPE_HARDWARE showed the elevation shadow, but upside down.


Always use named functions

Although v8 engine infers function names from variable definition (const x = function() {}, x is infered) we could use explicit function names to make debugging better.
An eslint rule might be used as well.

closing as is less priority


Example code endpoint has been moved

The original NWS weather alerts endpoint has been moved. The current example code will fail unless a valid NWS alerts ATOM feed is provided as input. It looks like NWS changed the feed's security in November 2016 and access to the new endpoint will need to be requested. https://alerts.weather.gov/#methods-get-cap-messages

resolved, must include User-Agent header!


Automatically Cache Calls To ParseMath

If someone calls ParseMath on a particular formula with the same variables+values then we should just pull out a cached result instead of recalculating. That helps alleviate lag caused by calling the same thing non-stop (like in a UI) without requiring the original caller to cache the value themselves (they should, since that would potentially prevent populating the variables dictionary, which could be slow). Every consuming mod could then experience improvements automatically.
I'm thinking a Dictionary<string, Hashset> with the raw formula as the dictionary key and then the variable dictionary defining the hashcode for the Hashset, with the result as the stored object's value. Maybe a double dictionary, but the variable dictionary would have to be checked the hard way then for everything which would be slow. Then you check if the formula is cached in general, then try to find the object for that variable dictionary, then you just get the value off it. If this is slower than just computing the new value, then that will cause everything to actually slow down. Hence the use of fast-access O(1) collections.
This is inspired by the trouble that caching the rollout time for vessels may cause in KCT. It's not trivial to cache, and if I could just cache this within MagiCore it would definitely help with performance automatically with no KCT code changes. I'm sure KCT calls the same formula multiple times in other places too.

This is completed in 1.3.0 using a dictionary that has a key that's the post-variable-replacement formula and a value that's the cached result of it. This works no matter who is calling the formula and still works even if unused variables are changed. Some (still fairly small) formulas were taking about 5ms to parse and with this were using the cached value in <1 ms, so it should help a little with lag when a formula is called non-stop.


JAN 1. Modify immigration filters to account for Colorado changes to Medicaid/CHP+ for undocumented pregnant women and children

NOTE: THIS CHANGE SHOULD BE IMPLEMENTED AS OF JANUARY 1, 2025 TO ALIGN WITH EFFECTIVE DATE OF CHANGE IN LAW.

[ ] If all sub-filters are deselected, the top-level should also deselect. If a user unchecks the 3 bullets under Green Card, then. "Green card holders" should also deselect

[ ] Update last filter to say "Other lawfully present noncitizens with permission to live or work in the U.S. (includes DACA recipients, other rules may apply)

NEW BEHAVIOR

If a user selects individuals without lawful U.S. presence or citizenship, Other lawfully present noncitizens . . . , or Green card holders, then, we should see if any of the HH members selected pregnant or are under 19. If yes for either, show Medicaid & CHP+
Add Warning message to Medicaid & CHP+ IF filters were selected, to say, "Based on your citizenship filter criteria, the individuals in your household that could qualify for [Medicaid / CHP+], are those who are younger than 19 or pregnant.
Edit value shown based on individual eligibility

Modify immigration filters to account for Colorado changes to Medicaid/CHP+ for undocumented pregnant women and children. See notes at https://docs.google.com/document/d/16bmdy-fTn27csU5naNFvhGZnwNM-Ewl5KWlII7qlvrg/edit?usp=sharing.

Hi @sydney-devine and @kbottagaro - I just wanted to make sure this last change, "New Behavior" will only apply to CO as NC has not expanded medicaid coverage to pregnant people and children without lawful status. The only result for someone who is pregnant and without lawful immigration status in NC is emergency medicaid and no medicaid results for children without lawful immigration status in NC. If only we could be so lucky as CO! It's great your state did that. Thx

Hi @sydney-devine and @kbottagaro - I just wanted to make sure this last change, "New Behavior" will only apply to CO as NC has not expanded medicaid coverage to pregnant people and children without lawful status. The only result for someone who is pregnant and without lawful immigration status in NC is emergency medicaid and no medicaid results for children without lawful immigration status in NC. If only we could be so lucky as CO! It's great your state did that. Thx

We'll do it so it's only CO. Also, work on this won't start til after Thanksgiving.

Thank you!

@sydney-devine The last task ("Edit value shown based on individual eligibility") was moved to https://github.com/Gary-Community-Ventures/benefits-api/issues/655

@sydney-devine The last task ("Edit value shown based on individual eligibility") was moved to Gary-Community-Ventures/benefits-api#655

ty


Integrate RPC in CLI

Resolves #1221
Added RpcServerOpts.

Codecov Report

Merging #1227 (7e5554b) into main (314ea08) will increase coverage by 0.66%.
The diff coverage is 0.00%.

:mega: This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more
@@            Coverage Diff             @@
##             main    #1227      +/-   ##
==========================================
+ Coverage   75.53%   76.19%   +0.66%
==========================================
  Files         339      339
  Lines       37674    37709      +35
==========================================
+ Hits        28456    28733     +277
+ Misses       9218     8976     -242

Flag
Coverage Δ

unit-tests
76.19% <0.00%> (+0.66%)
:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

bin/reth/src/lib.rs
4.54% <0.00%> (-5.46%)
:arrow_down:

bin/reth/src/node/mod.rs
0.00% <ø> (ø)

crates/net/network/src/manager.rs
49.24% <0.00%> (-0.19%)
:arrow_down:

crates/net/network/src/session/active.rs
86.35% <0.00%> (-0.15%)
:arrow_down:

crates/primitives/src/transaction/mod.rs
89.71% <0.00%> (+0.42%)
:arrow_up:

crates/net/network/src/listener.rs
83.72% <0.00%> (+2.32%)
:arrow_up:

crates/transaction-pool/src/pool/txpool.rs
61.45% <0.00%> (+2.62%)
:arrow_up:

crates/transaction-pool/src/pool/pending.rs
63.96% <0.00%> (+2.70%)
:arrow_up:

crates/transaction-pool/src/test_utils/mock.rs
62.05% <0.00%> (+10.28%)
:arrow_up:

crates/rpc/rpc/src/eth/api/mod.rs
10.44% <0.00%> (+10.44%)
:arrow_up:

... and 8 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

The parse test works but the implementation isn't clean, so any ideas to make it better is appreciated.

Just saw your impl for the test & I'm slightly pissed at mine.

I had to read the docs myself, because there's no easy way of doing this with Args trait directly.


Missing icons in some processors

Happens only on copy&paste, then only the first processor appearing in a pipeline gets the icon.


Provide a cluster worker pool

closes #66

@pioardi @jerome-benoit OH MY GOD IT WORKS :tada:

@pioardi How should I structure the tests folder?
Maybe also like the src folder?

@pioardi @jerome-benoit This PR is ready for a first review
But I need to write tests... You could potentially also help me with that
But before moving to much within the tests folder, I would prefer to have at least a little bit of feedback of the current state of the PR

Another thing that would be great is provide an example of using a ClusterPool...and within each process forked create a ...ThreadPool.... 🥇

@pioardi I added tests for cluster pool now
One test is failing :warning: Fixed cluster pool test suite => Verify that error handling is working properly
But I think that this is indeed correct!
Because the cluster error worker can not return an js-Error
'use strict'
const { ClusterWorker } = require('../../../lib/index')

function error (data) {
  throw new Error(data) // We are on another real CPU-core, we cannot return Error, because it's an js thing
}

module.exports = new ClusterWorker(error, { maxInactiveTime: 500 })

@pioardi I added tests for cluster pool now
One test is failing ⚠ Fixed cluster pool test suite => Verify that error handling is working properly
But I think that this is indeed correct!
Because the cluster error worker can not return an js-Error
'use strict'
const { ClusterWorker } = require('../../../lib/index')

function error (data) {
  throw new Error(data) // We are on another real CPU-core, we cannot return Error, because it's an js thing
}

module.exports = new ClusterWorker(error, { maxInactiveTime: 500 })

Then how we will handle errors?

A lot of tests seems to be duplicated, we could need a strategy to avoid duplications and simplify maintenance ( abstract class could help also on this )

Then how we will handle errors?

That is the question! What do you think / have in mind?

I found a solution! You can throw plain literals. And because these are not js-Errors but primitive data, you can receive them on the other side

I found a solution! You can throw plain literals. And because these are not js-Errors but primitive data, you can receive them on the other side

I think we will need to work on the cluster-worker.ts file , the worker can send messages to the master it should be really similar to what we did for other pools

I will try to take a more deep look into that in the meanwhile you can go ahead with your potential solution yeah

@pioardi Could you have a look into the tests? Some are randomly failing because GitHub Actions doesn't have the power to handle it stable
On my local machine (i9-10940X) it works perfectly ^^'

@pioardi Could you have a look into the tests? Some are randomly failing because GitHub Actions doesn't have the power to handle it stable
On my local machine (i9-10940X) it works perfectly ^^'

I was looking the same, on my local machine mac book pro i7 they are working, maybe we just need to increase the timeout let me take a look.
I looked at how you implemented the error handling and is ok, similar to the thread pools and sending a message to the main thread nice job on that.

I increased the time to wait of the tests, unfortunately we cannot predict or realize this in a more stable way ( we will need to have custom runners for CI ).
We could want to send an event when a new worker/thread  is created or terminated to handle this better

Preferable I would like to merge both open branches now (this one and #91)
So then we can start working on abstraction

A lot of tests seems to be duplicated, we could need a strategy to avoid duplications and simplify maintenance ( abstract class could help also on this )

Yes, a lot of code is duplicated. But I guess the goal is to factorise it later on.

@jerome-benoit When this branch is merged, do you want to try the abstraction?
So you have also something to work on :wink:
It's getting late for me and I worked the whole day on this branch
So I think I will take a break now and watch some YouTube / play some games

please check #100 just sharing a method for a better error handling. It is only a draft you can wait me to work better on that tomorrow or start from there from your own as you prefer :)

@Shinigami92  I am going to push the Error handling on this branch, do you prefer a direct commit or a PR into this branch ?

Error handling pushed @Shinigami92

I'm getting some messages while running Pioardi:Static:ClusterPool-benchmark
(node:131887) MaxListenersExceededWarning: Possible EventEmitter memory leak detected.
    10001 message listeners added to [Worker]. Use emitter.setMaxListeners() to increase limit

Also I don't know why, but I need to use relative path from root folder for the worker path in cluster pools :thinking:
Otherwise I get errors of path/to/poolifier/worker.js not found

Also, whats with the myBench.js? I think this was a legacy file? Can we remove it? Do we need somthing from it? Where can we run it?

Do we need to add workerpool to the tests for comparison?

General shape is good, I noticed that there is few coverage on some new files could you please check the coverage percentage? I would love to handle coverage into this PR, if you need help I can add some test on the next couple of days.

Only thread-worker file has really low coverage, but this was also before.
It's just relatively more now because there are generally more files.
Could you add more tests for this file? Or should we ignore it from the coverage, because it's only the worker?

Another thing that we should add ( this one also into another PR ) is the doc that explain the diffs between cluster and thread pools and when we suggest or not to use one over another

Sounds good, but sadly I cannot do this because I don't have the knowledge for that. I'm a frontend developer ^^ And I mostly write code in a quality way and just structured. But I don't know the depths behind such code/implementation.

De-duplication of code will be made in #111

Could you add more tests for this file? Or should we ignore it from the coverage, because it's only the worker?

We should not ignore it, is an important piece of the puzzle :) I can try to add more tests on the next days sure.
For the diffs and recommendations I can also do that.
I am concerned from the amount of code duplication into this PR https://github.com/pioardi/poolifier/pull/92/checks?check_run_id=1871793697
NB: I  integrated SonarCloud with this repo but I need to configure it better

I am concerned from the amount of code duplication into this PR #92 (checks)

This PR (92) is already REALLY big and I think it would be better and more clear for a review to de-duplicate the code in a new PR (issue #111)

@pioardi @jerome-benoit If you promise that this PR is in a stable stage and would be merged in this state, I can create a branch issue-111-encapsulate-logic-of-cluster-and-thread-worker-pool based on issue-66-provide-a-cluster-worker-pool
But it will be REALLY chaotic if more changes are coming into PR-92 before we merged PR-(issue-111) :warning:

I'm fine by doing the factorisation later on because of GitHub that enforce a code flow with some kind of atomicity at the PR level.
I'm more used to PR that contains a lot of changes and the atomicity is at the commit level but GitHub is not friendly with that kind of PR.

I did a rebase so the history within this PR as such is a bit more structured

I will start now working on further things based on this PR
:warning: PLEASE DO NOT COMMIT ANYTHING MORE INTO THIS PR DIRECTLY :warning:
When we are finished with the work on PRs based on this PR, we should squash-merge this PR first and then re-base the depending branches onto master
This way we will have a better history in our master branch and can reference more easily e.g. in the changelog the relevant squash-commits

Another thing that would be great is provide an example of using a ClusterPool...and within each process forked create a ...ThreadPool....

@pioardi If not already done, could you create an issue for that? Add example for ThreadPool within ClusterPool?

@pioardi this PR needs an approve, so I can squash-merge it by myself :slightly_smiling_face:

@Shinigami92  yes I don't see active comments on this PR excluding the one on code duplications.
Since you are already working on that, I am ok to proceed with this PR


typeinfo mismatch in separate Python extension modules that both produce Awkward Arrays

This issue is to document test failure reported in https://github.com/scikit-hep/awkward-1.0/issues/207#issuecomment-612451163 . From the looks of it, this is an instance of https://github.com/pybind/pybind11/issues/912.
Context
Everything presented here is done with MacOS 10.14 + nixpkgs (5c72e8416984879bd7bd884a392b7c915c28171a).
# clang -v
clang version 7.1.0 (tags/RELEASE_710/final)
Target: x86_64-apple-darwin18.7.0
Thread model: posix
InstalledDir: /nix/store/xbd074qa4p62sjbcmkvbzrb1d4zcj79n-clang-7.1.0/bin
# ld -v
@(#)PROGRAM:ld  PROJECT:ld64-450.3
BUILD 08:28:49 Apr  2 2020

The following options are unset, which, I guess, is the default.
_LIBCXX_DYNAMIC_FALLBACK
_LIBCPP_HAS_NONUNIQUE_TYPEINFO
_LIBCPP_NONUNIQUE_RTTI_BIT

dlopen flags:
>>> import sys; sys.getdlopenflags() == os.RTLD_NOW
True

Steps to reproduce
>>> import awkward1; awkward1.from_json("[1,2,3]", highlevel=False)[0]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
RuntimeError: missing boxer for Content subtype

Analysis
What happens under the hood is awkward1.from_json is calling into awkward1._io.fromjson
https://github.com/scikit-hep/awkward-1.0/blob/77c3e7ff7304734f7e917c9223adf473eace521b/src/awkward1/operations/convert.py#L418
which allocates an instance of ak::NumpyArray:
https://github.com/scikit-hep/awkward-1.0/blob/77c3e7ff7304734f7e917c9223adf473eace521b/src/libawkward/builder/Int64Builder.cpp#L61-L68
one part I don't understand is how it is able to return ak::Content:
https://github.com/scikit-hep/awkward-1.0/blob/77c3e7ff7304734f7e917c9223adf473eace521b/src/python/_io.cpp#L20-L25
Because pybind should not know about ak::Content as a python class without "awkward1.layout" which defines it in
https://github.com/scikit-hep/awkward-1.0/blob/77c3e7ff7304734f7e917c9223adf473eace521b/src/python/layout/content.cpp#L807-L810
When we call into __getitem__ we definitely get into awkward1.layout
https://github.com/scikit-hep/awkward-1.0/blob/77c3e7ff7304734f7e917c9223adf473eace521b/src/python/layout/content.cpp#L570-L572
and that somehow is dispatched to NumpyArray::getitem_at_nowrap in the "awkward1._io" library. There it allocates another NumpyArray:
https://github.com/scikit-hep/awkward-1.0/blob/77c3e7ff7304734f7e917c9223adf473eace521b/src/libawkward/array/NumpyArray.cpp#L776-L783
That we then attempt to box (why do even need the box, didn't we just successfully return ak::Content from the awkward1._io.fromjson? what happens with the ownership of content?)
https://github.com/scikit-hep/awkward-1.0/blob/77c3e7ff7304734f7e917c9223adf473eace521b/src/python/layout/content.cpp#L12-L25
but "awkward1.layout" fails to recognise an instance of NumpyArray and finally fails:
https://github.com/scikit-hep/awkward-1.0/blob/77c3e7ff7304734f7e917c9223adf473eace521b/src/python/layout/content.cpp#L121-L124
Inspecting typeinfo for the classes I found that:

Both typeid(*content.get()).name() and typeid(ak::NumpyArray).name() have the same mangled typeinfo name "N7awkward10NumpyArrayE"
typeid(*content.get()).hash_code() differs from typeid(ak::NumpyArray).hash_code()
typeid(*content.get()) != typeid(ak::NumpyArray)

Now, in principle, some implementations may choose to compare type_info instances by comparing their names only. [1][2] I've tried enabling that, it has some interesting behaviour depending on the value of _LIBCPP_NONUNIQUE_RTTI_BIT. For small values (like 0xF) there is no apparent difference, for very large values it crashes in the dynamic loader, for an intermediate value I get an error:
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "lib/python3.7/site-packages/awkward1/__init__.py", line 16, in <module>
    import awkward1.layout
ImportError: generic_type: type "IndexU8" is already registered!

There is also _LIBCXX_DYNAMIC_FALLBACK [3].
Inspecting the symbols
The next logical step was to look for a duplicate symbol for the typeinfo for akward::NumpyArray. We first examine the object files to see if different typeinfo's are present in different translation units.
Using nm:
# find . -name "*.o" | xargs nm -o | grep N7awkward10NumpyArrayE | c++filt -_ | grep "typeinfo for aw"
./_io.dir/src/python/_io.cpp.o:                  U typeinfo for awkward::NumpyArray
./layout.dir/src/python/layout/content.cpp.o:                  U typeinfo for awkward::NumpyArray
./awkward-objects.dir/src/libawkward/array/NumpyArray.cpp.o:<PHONE_NUMBER>02ce90 S typeinfo for awkward::NumpyArray

Or objdump:
# find . -name "*.o" | xargs objdump -tC | grep 'typeinfo for awkward::NumpyArray'
0000000000000000 g       01 UND    00 0000 typeinfo for awkward::NumpyArray
0000000000000000 g       01 UND    00 0000 typeinfo for awkward::NumpyArray
000000000002ce90 g       0f SECT   08 0000 [.const_data] typeinfo for awkward::NumpyArray

This looks very reasonable - only one translation unit has an instance of the typeinfo. The undefined symbol in ./_io.dir/src/python/_io.cpp.o can be easily removed by dropping make_fromroot_nestedvector, but that doesn't help the problem.
Now we take a look at the final python libraries:
# find result/ -name "*.so" | xargs nm -o | grep N7awkward10NumpyArrayE | c++filt -_ | grep "typeinfo for aw"
result/lib/python3.7/site-packages/awkward1/layout.cpython-37m-darwin.so:<PHONE_NUMBER>4acf60 S typeinfo for awkward::NumpyArray
result/lib/python3.7/site-packages/awkward1/_io.cpython-37m-darwin.so:<PHONE_NUMBER>205440 S typeinfo for awkward::NumpyArray
result/lib/python3.7/site-packages/awkward1/types.cpython-37m-darwin.so:<PHONE_NUMBER>2499e0 S typeinfo for awkward::NumpyArray
# find result/ -name "*.so" | xargs objdump -tC | grep 'typeinfo for awkward::NumpyArray'
0000000000000000      d  20 GSYM   00 0000 typeinfo for awkward::NumpyArray
00000000004acf60 g       0f SECT   0d 0000 [.const_data] typeinfo for awkward::NumpyArray
0000000000000000      d  20 GSYM   00 0000 typeinfo for awkward::NumpyArray
0000000000205440 g       0f SECT   0d 0000 [.const_data] typeinfo for awkward::NumpyArray
0000000000000000      d  20 GSYM   00 0000 typeinfo for awkward::NumpyArray
00000000002499e0 g       0f SECT   0d 0000 [.const_data] typeinfo for awkward::NumpyArray

I assume, "GSYM" supposed to denote a global symbol, not sure how loader deals with the three available pieces.
This is how symbol tables look for the *.so files in the awkward1-0.2.12-cp37-cp37m-macosx_10_9_x86_64.whl:
# find . -name "*.so" | xargs nm -o | grep N7awkward10NumpyArrayE | c++filt -_ | grep "typeinfo for aw"
./layout.cpython-37m-darwin.so:<PHONE_NUMBER>443738 S typeinfo for awkward::NumpyArray
./_io.cpython-37m-darwin.so:<PHONE_NUMBER>20ee60 S typeinfo for awkward::NumpyArray
./types.cpython-37m-darwin.so:<PHONE_NUMBER>245f80 S typeinfo for awkward::NumpyArray
# find . -name "*.so" | xargs objdump -tC | grep 'typeinfo for awkward::NumpyArray'
0000000000443738 g       0f SECT   0c 0000 [.const_data] typeinfo for awkward::NumpyArray
000000000020ee60 g       0f SECT   0c 0000 [.const_data] typeinfo for awkward::NumpyArray
0000000000245f80 g       0f SECT   0c 0000 [.const_data] typeinfo for awkward::NumpyArray

The problem is not seen with python3 and python3Packages.numpy from nixpkgs and awkward from the wheel:
>>> import awkward1; awkward1.from_json("[1,2,3]", highlevel=False)[0]
1

Summary
The problem itself seems to stem from having objects passed across multiple python libraries. Perhaps, it is triggered by some specific toolchain or libc++ version?
[1] https://reviews.llvm.org/D38599?id=118502
[2] https://dholmes215.github.io/c++/2017/06/30/fun-with-typeid.html
[3] https://github.com/llvm-mirror/libcxxabi/blob/master/src/private_typeinfo.cpp

Wow—this is incredibly detailed! Thanks!
In the pybind11 issue that you linked, there was a suggestion to set dlopen flags to os.RTLD_NOW | os.RTLD_GLOBAL instead of os.RTLD_NOW before loading the extension module. Does this work as a work-around?
In #209, I'm combining all of Awkward's extension modules into one, but this problem would reemerge for dependent projects written in C++ with Python interfaces on MacOS. Based on your description of the above, I think this would be a problem only when dependent project produce arrays (though that's probably the more important use-case for dependent projects).
To answer your question,

That we then attempt to box (why do even need the box, didn't we just successfully return ak::Content from the awkward1._io.fromjson?)

The motivation for box is that some outputs are not Content subclasses on the Python side. A zero-dimensional NumpyArray should become a Python number and ak::None should become Python None. I don't think JSON → Awkward for only a number or only JSON null are supported; if so, those cases would require a box.
On the other hand, maybe stronger boxing can protect the arrays across boundaries... In our design, these array nodes be immutable (#176 and #177) and are inexpensive to copy (compared to the array buffers). If this is a problem in the future, I could add a protocol to replace the ak::Content subclasses with generic objects (not copying the buffers, but everything else), pass them over the boundary, and reconstruct ak::Content subclasses on the other side. Then each library—Awkward and its dependent projects—would each have its own private symbols, though each dependent project would have to know about this stronger boxing/unboxing protocol.

Cross-linking some discussion from the ITK library dealing with this issue a few years ago (it looks like you've already gone down much the same path in 209, but could help w/ the separate module issue here):

https://insightsoftwareconsortium.atlassian.net/browse/ITK-3490 (esp the linked article:  http://www.russellmcc.com/posts/2013-08-03-rtti.html)
https://review.source.kitware.com/%23/c/21805/7/
https://review.source.kitware.com/%23/c/21755/6/

In the pybind11 issue that you linked, there was a suggestion to set dlopen flags to os.RTLD_NOW | os.RTLD_GLOBAL instead of os.RTLD_NOW before loading the extension module. Does this work as a work-around?

I did not check, but I believe it would work. Enabling RTLD_GLOBAL is bound to cause some obscure platform-dependent bugs in the future, so I don't think we need to investigate this.

I don't know how much of this is still valid, but since this looks most useful as a search result, I'm going to close it.


Feature/issue 244 business logic

@rasmus-storjohann-PG this is looking good, can you do a rebase and fix up your commit messages too?


Add a changelog

I've been thinking of incorporating use of generated changelogs for all my projects.
This https://github.com/yoshuawuyts/changelog seems like an awesome tool. Maybe we use that?

Nice! Are we ready for a version 1.0.0 release?

I think so yeah

Great ... I'll make a pull request adding a changelog generated using the above mentioned CLI tool

I'm getting an error

I have a feeling it's due to us not tagging a release in Git. I think the tool uses that to make the changelog.

I see ... how exactly do you make a git version tag? Also, I'm pretty sure that I can't make the tag ... you'll probably have to do it

Will take a look in a bit

Thank you!

Any luck?

It turns out that you can't create a changelog without details in a Cargo.toml ... I've submitted a feature request (yoshuawuyts/changelog#25)


Show player prices on transfer list

Hi, can you show the player prices on transfer list please, I see in the code it only works in club and in transfer search

I'll take a look this evening. Thanks for reporting.

Hi, also, I think the player price showing has broken.
the platform stuff has broken it, is it LCCost or something is undefined (cant remember right now)

Can you provide an error log and state which version of the script you are using? Also which platform are you using?
It is LCPrice and should be available on every correct response from Futbin.

Alright champ, give me a second, ill be back with an answer in a min

Uncaught TypeError: Cannot read property 'LCPrice' of undefined
at Function.onload (eval at E_c (:3:298), :53:180)
at :2:479
at eval (eval at exec_fn (:2:27), :31:101)
So the part that is undefined is the platform prices part call.
Using v0.1.3, was working before the update to that.

Alright, what is the response of this command in the developer tools console: repositories.User.getCurrent().getSelectedPersona().platform?
If you are using Chrome, on the right side you can see a link next to the error, it looks like `VM....:##'. That will bring you to the exact location where the exception occurs. You can set a breakpoint on that line and try to determine where it goes wrong, because I can't reproduce it here.

"XBO" is returned, so maybe that's been changed.
The error point is at
target.append('
Futbin BIN' + data[playerId].prices[platform].LCPrice + '
');

Oh, btw, I'm an angular 4 and node developer, and happy to help on any of this, I could fix the above myself, but I'm currently at work. In the future ill be sure to fix bugs and add features (I'm particularly interested in marking players that are currently usable in SBC's, based on SBC squads in futbin etc :)

Oh, I made a mistake in the update checking the xbox platform (copy/paste error). Will fix that bug immediately!
Any help is greatly appreciated.

The prices aren't shown on the Transfer List because it doesn't trigger the DOMNodeInserted event in the same way as the other screens. Feel free to do a PR for this.


Fix missing lock fields in KeyError and forbid CheckTxnStatus pushing async commit locks

I've tested locally that with this fix, the tests in https://github.com/pingcap/tidb/pull/20073 can pass.

@coocood PTAL

LGTM


[OSF-4958] get_absolute_url() for nodes should choose most appropriate representation

The only use case for this right now is registration nodes. Node.get_absolute_url() now checks Node.is_registered and switches it's return value based on that.
As an added bonus I threw in changing the Node Serializer's registrations relationship to point to the new registrations endpoint.
This PR is based off of https://github.com/CenterForOpenScience/osf.io/pull/4444. It takes advantage of some of the routes and views defined within. It should be held out until 4444 is merged or they should be merged as a team.

Good to merge when the build is passing.


when try to make: Error: spawn node-gyp ENOENT

Describe the bug
Siren building fails
To Reproduce
Steps to reproduce the behavior:

clone
make

Expected behavior
build completed successfully
Screenshots
stable ✔ $ make
yarn && yarn dev
yarn install v1.22.19
[1/4] Resolving packages...
[2/4] Fetching packages...
warning<EMAIL_ADDRESS>The engine "pnpm" appears to be invalid.
[3/4] Linking dependencies...
warning " ><EMAIL_ADDRESS>has incorrect peer dependency<EMAIL_ADDRESS>|| ^17.0.0".
warning " ><EMAIL_ADDRESS>has unmet peer dependency "<EMAIL_ADDRESS>warning " ><EMAIL_ADDRESS>has unmet peer dependency "<EMAIL_ADDRESS>warning " ><EMAIL_ADDRESS>has incorrect peer dependency<EMAIL_ADDRESS>warning " ><EMAIL_ADDRESS>has incorrect peer dependency<EMAIL_ADDRESS>warning "react-scripts > eslint-config-react-app ><EMAIL_ADDRESS>has unmet peer dependency "<EMAIL_ADDRESS>warning "react-scripts > eslint-config-react-app ><EMAIL_ADDRESS>has unmet peer dependency "<EMAIL_ADDRESS>warning "@storybook/addon-essentials > @storybook/addon-docs ><EMAIL_ADDRESS>has unmet peer dependency "<EMAIL_ADDRESS>warning "@storybook/addon-actions ><EMAIL_ADDRESS>has incorrect peer dependency<EMAIL_ADDRESS>|| ^17.0.0".
warning " ><EMAIL_ADDRESS>has unmet peer dependency "<EMAIL_ADDRESS>warning "@storybook/addon-essentials > @storybook/addon-docs ><EMAIL_ADDRESS>has incorrect peer dependency<EMAIL_ADDRESS>|| ^17.0.0".
warning " ><EMAIL_ADDRESS>has unmet peer dependency "@babel/core@*".
warning " ><EMAIL_ADDRESS>has unmet peer dependency<EMAIL_ADDRESS>warning "@storybook/react ><EMAIL_ADDRESS>has unmet peer dependency "<EMAIL_ADDRESS>warning "@storybook/react ><EMAIL_ADDRESS>has incorrect peer dependency<EMAIL_ADDRESS>|| ^15.0.1 || ^16.0.0 || ^17.0.1".
warning "@storybook/react ><EMAIL_ADDRESS>has incorrect peer dependency<EMAIL_ADDRESS>|| ^15.0.1 || ^16.0.0 || ^17.0.1".
warning "@storybook/test-runner ><EMAIL_ADDRESS>has unmet peer dependency<EMAIL_ADDRESS>warning "@storybook/test-runner ><EMAIL_ADDRESS>has unmet peer dependency<EMAIL_ADDRESS>warning " ><EMAIL_ADDRESS>has unmet peer dependency "<EMAIL_ADDRESS>warning " ><EMAIL_ADDRESS>has unmet peer dependency "<EMAIL_ADDRESS>warning " ><EMAIL_ADDRESS>has unmet peer dependency "<EMAIL_ADDRESS>[4/4] Building fresh packages...
[1/6] ⠠ lzma-native
[6/6] ⠠ electron
[-/6] ⠠ waiting...
[4/6] ⠠ electron-winstaller
error /eth/siren/node_modules/lzma-native: Command failed.
Exit code: 1
Command: node-gyp-build
Arguments:
Directory: /eth/siren/node_modules/lzma-native
Output:
node:events:495
      throw er; // Unhandled 'error' event
      ^

Error: spawn node-gyp ENOENT
    at ChildProcess._handle.onexit (node:internal/child_process:284:19)
    at onErrorNT (node:internal/child_process:477:16)
    at process.processTicksAndRejections (node:internal/process/task_queues:82:21)
Emitted 'error' event on ChildProcess instance at:
    at ChildProcess._handle.onexit (node:internal/child_process:290:12)
    at onErrorNT (node:internal/child_process:477:16)
    at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
  errno: -2,
  code: 'ENOENT',
  syscall: 'spawn node-gyp',
  path: 'node-gyp',
  spawnargs: [ 'rebuild' ]

*** Error code 1

Stop.
make: stopped in /eth/siren

Desktop (please complete the following information):

OS: FreeBSD 14.1-RELEASE-p1 FreeBSD 14.1-RELEASE-p1 releng/14.1-n267682-0892dff10444 amd64
Browser
Version

Additional context
stable ✔ $ yarn --version
1.22.19
stable ✔ $ node --version
v18.20.3

@ip75 we have recently announced a new siren release candidate for v2.0.0. where we have completely removed storybook dependencies and no longer build an electron app. you can try this docker image or build from unstable branch and follow the instructions in the readme there.

yes. You are right. It is ok on unstable branch. Thanks.


Implement backpressure in the Netty implementation of NioServer

With support for large blobs, POST requests can be very large and data can arrive at a rate faster than the frontend can process.
To avoid memory pressure, data should be read from the channel only when frontend is ready to receive more. Netty provides support for on-demand reading from the channel via the auto-read feature (docs).
This support can be built into NettyRequest. At a high level, when a POST is detected, a pre-configured number of chunks are loaded and auto-read is set to false. Channel#read() calls are triggered when callbacks arrive from the AsyncWritableChannel(because "empty" slots become available). Once the request is complete, auto-read is set to true again.
To support all of this, the constructor of NettyRequest (and its children) has to be changed to receive a Channel and the tests have to be changed accordingly. The production code should be simple enough to change but the test code might take longer.
Acceptance Criteria:

Build intelligence into NettyRequest to apply backpressure and read from channel on-demand so that there is no memory pressure in case the Router needs time.
Verify functionality via tests that clearly demonstrate and exercise the application of backpressure.

Is this newbie?

Yeah I think so. It is very self contained and requires understanding some components. It will require some help but I think it's a good newbie project. What do you think?

What is the total work involved? 2 weeks? 3 weeks? A newbie issue should be something that focuses on a narrow part of Ambry and should take 1 week to fix.

Shouldn't take more than a week end-to-end including tests but requires some understanding of Netty. I can remove the newbie tag for now.


Tips of scalable annotation rendering for developers

As someone who needs to display (tens of) thousands of elements (polygons, rectangles, points) in a single slide, I am creating this issue to have documentation of the discussion on how to effectively display numerous polygons in a slide. This may prove to be useful to future users and is not immediately obvious, hence the motivation to write this discussion for my future self (hello there!) and others.
1. Some definitions and FAQs
- What is an Annotation?
An annotation, aka annotation "document", is the single unit containing multiple elements (points/rectangles/lines/polygons/etc). When communicating with the HistomicsTK server using the annotation endpoint of the API, this is the indivisible unit used for sending and receiving data in a JSON-like format. The distinction between annotation documents, groups, and elements, is a bit suboptimal from a user standpoint, but is a useful abstraction for efficiency for developers (see discussion in #625 , #636). So even future changes to this model are likely to just impact display (i.e. abstract things away from the UI) but not the underlying principles.

- What is an Element?
An element, aka annotation "element". This is a single points/rectangles/lines/polygons/etc. Elements belonging to the same "document" are stored as a list as part of the "Elements" attribute of the document.
- Is there a limit to the number of loaded annotation documents?
Yes! To avoid crashing the browser, there is a maximum number of annotation documents that are sampled for display. This number is currently set to 5,000, see #670 . This means that if your algorithm has saved more than 5,000 annotation documents, the end user will only be able to see 5,000 documents when they load the slide.
- Is there a limit to the number of loaded annotation elements?
Yes! If the number of elements in the documents is more than 2,000,000 then the smallest elements (as determined by bounding box diagonal) are displayed as circles, see #676 and #731 . Of course, if you have more than 5,000 documents in your slide then by definition some elements will be missing since only 5,000 documents are displayed.
2. Some efficiency considerations - (Also see discussion at #636)
- Which elements are more efficient to render?
The basic types of elements include: point, rectangle, polygon, and line. The most efficient elements to render are, in order of increasing render efficiency:
Filled polygon -> Unfilled polygon (no opacity) -> Rectangle -> Line -> Point

Obviously, the more vertices of the polygon the less is the rendering efficiency.
- What are the least performant operations for the end-user?

Rendering thousands of filled polygons with numerous vertices (#592 ).
Using the transparency slider - this re-renders the polygons, so does not perform well when there are thousands of filled polygons, see #672
Right clicking and editing an element when there are thousands of RENDERED elements (see #533 ). For example, say the slide is associated with 500 annotation documents, each of which contains 2,000 annotation elements. If the user clicks the 'eye' icon to render ALL 500 x 2000 = 100,000 elements, then right clicking on any single element to edit will be very slow. If the user just chooses to render a couple of documents (i.e. only 1,000 elements) at a time, then there is no performance issue.

3. Long story short, what should I do?
- "Spread out" your annotation elements over many annotation documents
Of course, keeping in mind that you do not exceed the maximum limit of 5,000 documents. This makes sure the server sends and receives 'bite-sized' (in a non-computer-science sense) packets of data, which is good for everyone. Here's why:

It is good for pushing data using the POST or editing data using the PUT endpoints using the API without getting a request time out.
It makes sure the user can edit the individual elements without problems. Note that right clicking an editing an element using the UI uses the PUT endpoint to edit the elements attribute of the annotation document to which the to-be-edited element belongs. So even if the user wants to edit one single annotation element, the client sends a PUT request containing the entire new, edited annotation document. Is this optimal behavior? It could probably be improved, but for now, keep this in mind. (Also see discussion in #719).
It allows the user to display groups of elements at a time if the user really desires highly performant rendering, see #592 .

- Consider which elements to group together into the same document
This depends on the use case. For example, you may decide to group all elements from the same region-of-interest (or geographic location) into the same document so that the user can view all annotations in the same locality with one click on the "eye" icon. If there is no natural "region of interest" in your project and you just have tens of thousands of objects spread throughout the slide, consider grouping them by geographic proximity.
Alternatively, you may decide to have elements of the same group (eg tumor) to be in the same document. This relates to #698 and #592  . Again, even if you elect to use this option, make sure to spread things over many documents if possible.
- Minimize number of filled polygons
This goes without saying. Any polygon fill is rendered on the GPU and is re-rendered with the interactive mode, transparency slider, right click, multi-select, lasso select, etc.
- Minimize vertices per polygon
If your polygons are displaying algorithmic outputs, and they were originally part of a mask at, say. 40x magnification, consider resizing the source mask to a smaller magnification (say, 15x) before extracting the polygonal coordinates. If you don't expect the user to need a level of detail beyond 15x, only use enough vertices to be accurate at 15x.
- Make sure the end-user knows about efficiency bottlenecks
If your end user (eg pathology resident) needs to edit elements in a slide with hundreds of thousands of elements, make sure to let them know which things to do and not to do.
That's all I can think of, but happy to edit and extend this based on suggestions.

@manthey @cooperlab Any other tips to add? Also, do you think this belongs in the documentation somewhere, or is it best placed in the issues as it is now?

Ultimately, I think this is better somewhere in the docs.  When using a new project, I expect docs; I don't expect documentation in issues.
With the centroid and size PR, some of this isn't quite true any more.
Also, as a thought, one of the reasons we don't edit large numbers of elements is because we list all of the elements in the bottom right panel, and it would overwhelm the html DOM.  Ideally, we should handle these somewhat like the view -- show those that are non-centroids and allow them to be edited (this will take some work as we currently resave the entire element set instead of updating a few elements).

Thanks @manthey, this would definitely be very helpful if and when implemented.
I've incorporated this into the documentation in a branch I am finalizing for a pull request.

Incorporated into main docs.


Cannot specify nested BigQuery schema with BigQueryTableFieldSchema

BigQueryTableFieldSchema wraps around TableFieldSchema. However, it didn't provide an API to set nested schema fields for RECORD type like TableFieldSchema#setFields.
For example, the public dataset bigquery-public-data:samples.github_nested has a nested schema as follows.

With TableSchema/TableFieldSchema, we are able to specify payload field as
new TableFieldSchema()
  .setName("payload")
  .setType("RECORD")
  .setFields(List(
     new TableFieldSchema().setName("action").setType("STRING"),
     new TableFieldSchema().setName("after").setType("STRING"),
     ...
  ))

The wrapper class BigQueryTableFieldSchema didn't provide a similar API like setFields. Therefore, such nested schema cannot be provided to BigQueryOutputConfiguration.configure as a typed Java object.

jiayuanmasc@ Thanks for the feedback, will fix it soon.


The boilerplate project is outdated and does not work well with the new version

After your update, 2.2.0, if someone is using your boilerplate project, it will stop working because there is an incompatibility with typescript-rest-swagger and multiple authenticators.   I've added the stack trace in this other bug #91. Can you please update the documentation with a relevant example on how to use the securityDefinitions in swagger.config.json and the Swagger UI with Bearer Authorization?
I have this in swagger.config.json:
"securityDefinitions": {
	"Bearer": {
		"type": "apiKey",
		"in": "header",
		"name": "Authorization"
	}
}

This produces the following Swagger UI authorization:

However, when I try any request in Swagger UI, the Authorization Header is not added: curl -X GET "http://localhost:8080/users/1/accounts?includeExpired=true" -H "accept: application/json"
I then updated to 2.2.0 and tried to add Server.registerAuthenticator(new MyCustomTokenAuth(), 'MyAuth'); and on my controller, I use @Security('ROLE_USER', 'MyAuth') but it seems that typescript-rest-swagger refuses to generate the swagger file with the following error:
node_modules/typescript-rest-swagger/dist/metadata/controllerGenerator.js:93
            scopes: d.arguments[1] ? d.arguments[1].elements.map(function (e) { return e.text; }) : undefined
TypeError: Cannot read property 'map' of undefined

Could you update your boilerplate example and the documentation to provide a clear example on how to achieve this? Thanks.

Done.  I opened a PR to typescript-rest-boilerplate:
https://github.com/vrudikov/typescript-rest-boilerplate/pull/10

Hi. I see the update but this doesn't answer the Swagger UI Authorization question. Does it work with the new version 2.2.0 or not yet? And if it's supposed to work, can you include an example? This is a very common use case. Also, there's a comment that the update is actually breaking.

Hi @evelyne24,
Yes, it works with version 2.2.0 (+), but you need the latest version of typescript-rest-swagger (1.0.3 at least).
I send a PR to typescript-rest-swagger showing an working example: https://github.com/vrudikov/typescript-rest-boilerplate/pull/12
I added two authenticators. Both of them using jwt with the secret: some-jwt-secret


Testing of Ingestion_Api for cQube_4.0 (Smoke-Testing)(S3)

This issue is not covered in v5.0 and needs to be analysed.


how to select device (webcam) for directshow

Hi,
when I am using the following code, I always get the output from the first device and not from the the selected cam. I have tested the option in VLC and it works.
var camoptions = new string[] { string.Format(" :dshow-vdev=\"{0}\"", "Microsoft® LifeCam HD-3000") };
myControl.MediaPlayer.Play(new Uri("dshow://"), camoptions);

PLEASE HELP !!!! Its urgent that I achieve this functionality.
Everything else works fine,
Thanks in advance.

Your problem is not a Vlc.Dotnet one but about VLC options syntax. You can find help on the VideoLAN Forums


Run.py

Hi,
I came across this scraper code a month ago and was able to run it successfully through Gitpod. I am trying again now and I cannot find the run.py file. What changed?
Thanks

I had the same issue and contacted them about it. I got this answer:
We have made the decision to transition our Tool from Python and Node.js versions to a desktop application. This change is aimed at providing you with a more user-friendly and convenient experience while utilizing our Tool.
Please be advised that the last available version of the Python and Node.js implementations is dated October 4, 2024. Should you wish to access this version, you may do so by visiting the following link: https://www.omkar.cloud/l/google-maps-scraper-pro-node-python/
To install and use this version, please follow the instructions provided in the README.md file included in above zip file.
We kindly request your understanding that, moving forward, all future updates and improvements will be exclusively available for the desktop version of our Tool. As a result, we will no longer provide updates or support for the Python and Node.js versions.
Alternatively, to use the Desktop Version, please follow these steps:

Install the Desktop App by visiting https://github.com/omkarcloud/google-maps-scraper#-requirements.
Activate your Pro version by visiting https://www.omkar.cloud/l/google-maps-scraper-pro/.

We greatly appreciate your continued support as we strive to enhance your experience with our Tool.

@rafalolhub I edited your Commend to hide links that should be available for Pro Users Only, In case some one needs those links, please contact<EMAIL_ADDRESS>


Add a new type of setting:   global

Settings are stored in the SETTING table in the database.    Its a nice design - nothing is ever deleted, instead we just mark a "is_deleted" flag and an "is_active" flag.  The actual setting information is stored as json in the 'values' column in the SETTING table.
Currently there are organization, receiver, and sender settings, and the structure corresponds exactly to Organization.kt, Receiver.kt, and Sender,kt, so that the standard Jackson object serialization can transform between json and Kotlin objects.
We need a new object GlobalSetting.kt, that can store global settings per environment -- my goal is to be able to have different settings in Prod, Staging, and Local.
For an example of the need, if you look down at the bottom of JurisdictionFilters.kt, you'll see a defaultCovid19QualityCheck --- I want that to appear in the global settings instead of hardcoded, so that I can have a different value for Staging and Prod.

Note that even tho we store json, we almost always send in yams - the ./prime settings can take yaml input, so I'm using that here.

Prior to doing the work, make sure you understand the Settings API code, found in SettingsFunction.kt and SettingsFacade.kt to do this task, as well as the code that can send settings from the CLI in SettingsCommands.kt.      The SettingsFunction in particular has a complex class structure that it took me a while to understand.

Your job will be to add a new kind of setting, global, and implement it in all those classes.

Let's start with just a simple setting:  the environement, which could be 'Prod', "Staging" or "Dev"

I'm waffling about how generic to make this.   For example:

- key:   environment
  value:  Staging
- key:  foo
  value: bar

Or
- environment: Staging
- foo: bar

Questions: do we just have single globalSetting object in the database, or do we allow any number of them?  If there's only one global setting obj allowed, how do we enforce that?  Maybe each globalSetting only has a single key/value pair?  And what if they mistakenly repeat the same key?    Use the defaultCovid19QualityFilter as an example, since its a more complicated object - we want to handle that case.
Tasks:

study and understand the settings code, table, CLI.  Play with the cli.
propose wise answers to all life's persistent questions, or at least, those listed above.   Discuss at Monday meeting if needed, or just with Jim.
Think about other thorny problems I haven't thought of
build, test, deploy
the follow-on work, which we can ticket separately, is to move that defaultCovid19QualityFilter to a setting.

A huge use case for this is to put the default Quality, ProcessingMode, Routing filters could go in 'global' instead of hardcoded in kotlin.


Accessing cursors in component

In version 1.x we could access cursor instance via this.cursors. Now, it was removed from the documentation. Why?
And how to access to cursor instance when we use HoC?

This was removed to enforce architectural good practices and to boost performance. You are now required to use actions to edit the tree.

Was searching on this for a little while; Would be good to have a section on differences between 1.x and 2.x outlining the major changes.

Swapping the arguments on branch/tree
Removing the wrapper components
Accessing cursors via context

This was removed to enforce architectural good practices and to boost performance

@Yomguithereal Could you elaborate on the performance aspect? Is the not performant to pass cursors around as props?

You should avoid to create too many cursors (say, creating one cursor per element of a long list is not advisable). Because even if they are cached as singleton per path, you still register some listeners and observers.

Is the not performant to pass cursors around as props?

This is not really an issue but I don't really see why you would like to do that. In my experience, components just want to consume data from your tree and trigger some actions but not having control over the underlying cursors.
In this regard it's actually quite similar to redux etc.

This is not really an issue but I don't really see why you would like to do that

So, let's say:

I pass around cursors as props in components
Somewhere up in component tree, I attach an update listener to a parent cursor like this:

parentCursor.on('update', function () {
  this.setState({...})
)

Then anytime, I call cursor.set() in any child component (which takes some child cursor as prop), the event handler will fire, and re-render the tree because of setState.

Sure. But this is more or less what the baobab-react helpers handle for you.

The problem I ran into with baobab-react is that if I need to modify a node in tree, the action handlers need to know the entire path of the node from the root. Is there a better way of doing that?
For example, if we look at actions.js in this, we are directly doing a tree.push('colors', color). So, if colors were, 4 level deep, that'd become a problem.
Redux seems handles that by reducer composition (combineReducers).
Is there a better way to handle that in baobab-react ?

Usually, I do the following: I often split my action files hierarchically and they all have sets of cursor paths that the actions can use. Or I use function composition to abstract this issue. Another solution is to pass the paths as the actions' arguments but I am not so found of this.
Redux is already more abstract and indeed handle this somewhat using combineReducers but lacks in flexibility, notably when you need to burst out of this abstraction (two sibling reducers handling somewhat dependent state is a bit of a pain in redux for instance).


Add Discord link to GitHub Readme

Feature Summary
More ways to join the community, maybe a discord button
Detailed Description
https://discord.gg/9vrAZ8xGyh
Maybe a button image added to the repo, too.

I can work with this.

@Tiansing tgst would be great! Use the link above and either add a text link next to the site links under the logo or a standalone “join button”. See what looks better!
Let me know if you have any questions getting to and running!

@seriouslysean can I have the discord server ID? in order for me to create the discord button

Hi @seriouslysean , I'm the one who's assigned to add discord link to Github Readme and I just want to request, if you are able to enable the server widget under server settings?  something like this:

for me to create the discord button similar to this:

@seriouslysean I have created a new PR #146 Thank you

Closed by #146


Re-open call for contributions: Update contribution guidelines, add PDF, and remove card

Context
Since the last time we opened a call for contributions, we have altered the contributing guidelines that we track in ScPCA-admin.
Problem or idea
We need to port over the up-to-date guidelines, once a few issues I'll set as blocking are completed, and reverse the steps in #430

[x] Update the contributing guidelines to reflect the most recent changes.
[x] Add a PDF version to /src/public/
[x] Remove the card on /contribute that says we're not accepting contributions
[x] Update config contribute_hsform to the form for contributing
[x] Add back the contribution banner on the home page

Solution or next step
I will complete the blocking items and then handoff to the Engineering team.

Note: We are going to go with the new style of banner that renders above the navigation elements.
ex:

Merged PRs #606, #607, #626.


Failed to load LDAP config file when using Jinja templated integer variable

When a Jinja templated variable is used as LDAP port value like this:
grafana_ldap:
  servers:
    host: "{{ ldap_server_name }}"
    port: "{{ ports.ldap }}"

this is parsed as a string in ldap.toml, resulting in the following error:
t=2019-09-06T14:26:47+0200 lvl=eror msg="Error while trying to authenticate user" logger=context userId=0 orgId=0 uname= error="Failed to get LDAP config: Failed to load LDAP config file: toml: cannot load TOML value of type string into a Go integer"
Possible solution:
in ldap.toml.j2
{% for k,v in grafana_ldap.servers.items() if k != 'attributes' %}
{%   if k == 'port' %}
{{ k }} = {{ v | int }}
{%   elif v in [True, False] %}
{{ k }} = {{ 'true' if v else 'false' }}
{%   else %}
{{ k }} = {{ v | to_nice_json }}
{%   endif %}
{% endfor %}

Sorry, this flew under the radar. Your fix seems good and I included it in #180.


Pyblish-pype toggle instance callback

Currently pyblish-pype does not support instanceToggled:
pyblish.api.register_callback(
    "instanceToggled", on_pyblish_instance_toggled
)

I expect there are arguments for on_pyblish_instance_toggled function, do you know what they expect?

Do we want pluginToggled?

Havent got/seen a use case for that, so no need to add it for my sake.

I see only one possible issue: It's possible to toggle instance outside GUI (e.g. inside plugin) and I can't much catch that...


bug: don't appear popupmenu also classical bottom cmdline or search

Did you check docs and existing issues?

[X] I have read all the noice.nvim docs
[X] I have searched the existing issues of noice.nvim
[X] I have searched the existing issues of plugins related to this issue

Neovim version (nvim -v)
NVIM v0.9.0
Operating system/version
macOS 13.1
Describe the bug
Input NoiceLog with some error as below:
Mon May 29 13:42:30 2023
....local/share/nvim/lazy/lazy.nvim/lua/lazy/core/cache.lua:108: EACCES: permission denied: /Users/apple/.cache/nvim/luac/%Users%apple%.local%share%nvim%lazy%nvim-notify%lua%notify%render%base.luac

stack traceback:
	[C]: in function 'pcall'
	...local/share/nvim/lazy/noice.nvim/lua/noice/util/call.lua:144: in function <...local/share/nvim/lazy/noice.nvim/lua/noice/util/call.lua:143>
	[C]: in function 'assert'
	....local/share/nvim/lazy/lazy.nvim/lua/lazy/core/cache.lua:108: in function 'write'
	....local/share/nvim/lazy/lazy.nvim/lua/lazy/core/cache.lua:261: in function 'load'
	....local/share/nvim/lazy/lazy.nvim/lua/lazy/core/cache.lua:160: in function <....local/share/nvim/lazy/lazy.nvim/lua/lazy/core/cache.lua:156>
	[C]: in function 'require'
	...
	[C]: in function 'xpcall'
	...local/share/nvim/lazy/noice.nvim/lua/noice/util/call.lua:149: in function 'try'
	...local/share/nvim/lazy/noice.nvim/lua/noice/view/init.lua:140: in function 'display'
	.../share/nvim/lazy/noice.nvim/lua/noice/message/router.lua:215: in function <.../share/nvim/lazy/noice.nvim/lua/noice/message/router.lua:155>
	[C]: in function 'xpcall'
	...local/share/nvim/lazy/noice.nvim/lua/noice/util/call.lua:149: in function <...local/share/nvim/lazy/noice.nvim/lua/noice/util/call.lua:134>
	[C]: in function 'pcall'
	...local/share/nvim/lazy/noice.nvim/lua/noice/util/init.lua:108: in function ''
	vim/_editor.lua: in function ''
	vim/_editor.lua: in function <vim/_editor.lua:0>

Steps To Reproduce

use / or :
will appear popupmenu
for a while, get some error

Expected Behavior
always popupmenu statue
Repro
require("noice").setup({
  lsp = {
    -- override markdown rendering so that **cmp** and other plugins use **Treesitter**
    override = {
      ["vim.lsp.util.convert_input_to_markdown_lines"] = true,
      ["vim.lsp.util.stylize_markdown"] = true,
      ["cmp.entry.get_documentation"] = true,
    },
  },
  -- you can enable a preset for easier configuration
  presets = {
    bottom_search = false, -- use a classic bottom cmdline for search
    command_palette = true, -- position the cmdline and popupmenu together
    long_message_to_split = true, -- long messages will be sent to a split
    inc_rename = false, -- enables an input dialog for inc-rename.nvim
    lsp_doc_border = false, -- add a border to hover docs and signature help
  },
})

You clearly can see a permission error. Nothing I can do about that obviously. Fix your setup


Fix example in README.md

new ButtonTemplate expects object as param, not single params

Coverage remained the same at 100.0% when pulling 1012d0f93d5683af897e3eae1862ec9f4c467b5b on SaadeMotion:patch-1 into f5a95b30477be446764f0227b045f544518d085f on rickydunlop:master.

Thanks!