Add support for bulk participant creation

Refs:
https://openscience.atlassian.net/browse/LEI-305
(and maybe) https://openscience.atlassian.net/browse/LEI-312
Companion to: https://github.com/CenterForOpenScience/exp-addons/pull/198
Purpose
Add support for bulk participant creation. Limit to 100 records per attempt.
Summary of changes

Add bulk support for account creation: send 100 records in 1 request instead of 100
Rely on JamDB bulk error reporting failure mechanism- instead of preemptively checking that a record exists already and guaranteeing 100 creations, let some fail and report how many succeeded. (may get a 200 status even if individual creation operations fail)

Testing notes

Create 1 record, check CSV
Create 1 record with a specified field containing a password, and make sure extra fields are saved to CSV file
Enter invalid batch size or study ID. Validation error text should not show initially, but should always show after the first time you click "create". Errors should display inline under the affected field.
Batch size should be limited to 100, and validation should show errors when appropriate. Try typing in a batch size of 0, 1, 1.5, 100, and 101.
Batch-create 2 records. Without reloading the page, create another batch of 2 participants. The second and final CSV file should have 4 entries (including the 2 from the first batch)
Try to create 2 records, but edit the code for createParticipants to add a third record ID that already exists. The response payload should report 2 successful creations and one null placeholder in the data section (for the 3 records sent to the server), with 2 null placeholders and one "Document already exists" value in "errors" .

Known issues

The UI appears to hang when creating a lot of records; this is not because of network requests. It seems that serialization is the slow step.

At the moment, we are forced to go through the serializer because that is where the logic to bcrypt passwords live. (and because we want the model to be the authority on how records should look on the server) Constructing a payload from scratch might get a performance gain, but we'd prefer other options first, including profiling what it is about serialization that is so slow.

Batch-create 2 records. Without reloading the page, create another batch of 2 participants. The second and final CSV file should have 4 entries (including the 2 from the first batch)

Why the change from the original behavior, where each batch was downloaded to a CSV file? I find it unintuitive to have to refresh the page each time I need the responses written to a new file.

Why the change from the original behavior, where each batch was downloaded to a CSV file? I find it unintuitive to have to refresh the page each time I need the responses written to a new file.

The refresh is a bit surprising and not an original change- was it like that before.
Basically, you could always click the button again to generate more records. You should get a CSV file every time you click create- the only intended difference is that now, your CSV file should reflect all the work you did on that visit to the page. (not just the most recent batch)
Under the hood, this is because bulk transitions are usually limited- you can't ask the server to do infinite work in a single request. In the OSF we limit it to 100 items per request, for example. The idea was that someone could respect that limit, but not have to track a bunch of extra CSV files and combine them on their own.

There is an issue with writing to the same CSV file when you change the extra fields. If I create records with the extra field "A", and then I delete that extra field and replace it with "B", the new CSV does not have the extra field "A" in it.

There is an issue with writing to the same CSV file when you change the extra fields. If I create records with the extra field "A", and then I delete that extra field and replace it with "B", the new CSV does not have the extra field "A" in it.

Hmm, that...  is probably a fair point for how CSV files are written. (it needs to get the list of fields from somewhere, and won't deal well if the rules change)
I'll simplify this down to only serializing the most recently created batch of records for now; we can revisit the added complexity required for improvements later.

@abought is it possible to change the participant ids in the downloaded csv from isp.accounts.<participant_id> to just the participant_id? After that, this is good to merge :octocat:

@abought is it possible to change the participant ids in the downloaded csv from isp.accounts.<participant_id> to just the participant_id? After that, this is good to merge :octocat:

I've pushed the changes- the csv writer now receives records again instead of pure JSON. As a result, headers are of the form id, extra.studyId, etc.
If it wouldn't trouble you too much, this could use an extra pair of hands poking one more time. I had to hack around ember a bit and create quasi-saved records, so I added some cleanup code (to make sure the broken account objects don't hang around the ember store later). It should clean up save errors from the ember inspector on save, and other accounts when you leave the page.

Looks good to me! I tested that an account with a participant ID that already exists cannot be created and gets cleared from the store. Also, remember to update the testing notes since each batch of participants is written to a separate CSV file.


ENH: Adding AMRFinderPlusAnnotation type

solves #85

Added new type SampleData[AMRFinderPlusAnnotations] to be used with SampleData[MAGs] annotation.
Added new type FeatureData[AMRFinderPlusAnnotation] to be used with FeatureData[Sequence] annotation.

Download test data:
PR-86.zip

Test it out!

qiime tools import --input-path sample_data --output-path amr_annotations.qza --type "SampleData[AMRFinderPlusAnnotations]"
qiime tools import --input-path feature_data --output-path amr_annotation.qza --type "FeatureData[AMRFinderPlusAnnotation]"

Hi @misialq
Can you merge this one for me too?
And can you remove those checks that are not needed anymore so i can merge the PRs by myself? The checks are still there also on newly created PRs.

Hey @VinzentRisch, I finally figured why all these check were always appearing on all of your PRs and fixed it - from now on, only the two should be visible. What I am wondering, though, is why the coverage report does not show up here (or anywhere else?) - can you please investigate? I'm merging this one in the meantime.

Oh, and let's stop using the `` ticks in commit names/PR titles - it breaks our CI 😅

On the topic of CI bugs. @misialq
I think revert commits also break the CI
see here: https://github.com/bokulich-lab/q2-amr/actions/runs/9807765273/job/27082173410?pr=86#step:4:12
Or is it something else?

Yeah, it seems so... my suspicion is that the additional quotation marks " in the revert commit message may have something to do with that...


PRO-2207: Add column attribute to ensure that creation time cannot be…

… updated
Before creating a pull request make sure that:

[ ] commit messages are meaningful and follow good commit message guidelines
[ ] README and other documentation has been updated / added (if needed)
[ ] tests have been updated / new tests has been added (if needed)

Please remove this line and everything above and fill the following sections:
JIRA link (if applicable)
Change description
Does this PR introduce a breaking change? (check one with "x")
[ ] Yes
[ ] No

Closing this PR as it is already merged from demo-> master


Add Switch Theme

Add a switch  for light & dark theme and improve design.

too big icon in browser resize or make size in css
bad typo in code miss a lot tab's
button toogle theme need be a litle more big align right


Doc: how to exclude tests in Maven build

For gradle the docs have:
tasks.withType(JavaCompile) {
  // remove the if condition if you want to run NullAway on test code
  if (!name.toLowerCase().contains("test")) {
    options.compilerArgs += ["-Xep:NullAway:ERROR", "-XepOpt:NullAway:AnnotatedPackages=com.uber"]
  }
}

It's not immediately obvious how to do this in a maven project. Please expand the maven docs to demonstrate.

If you want to skip tests, I believe that adding -DskipTests will bypass the Maven surefire plugin.
I'm not sure about the "verify" step, though.
More info:

https://maven.apache.org/surefire/maven-surefire-plugin/examples/skipping-tests.html
https://maven.apache.org/surefire/maven-surefire-plugin/test-mojo.html#skipTests

The question is not how to skip running the tests. Rather it is how to keep NullAway from checking the tests for null problems.

Oh, sorry for the misunderstood.

I'm not a Maven expert. @kageiit any ideas or thoughts on who would know
this? Basically need a way to tweak the "main" javac args but not those for
test code.
On Tue, May 8, 2018, 10:58 Jeanderson Barros Candido <
<EMAIL_ADDRESS>wrote:

Oh, sorry for the misunderstood.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/uber/NullAway/issues/162#issuecomment-387489101, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AALyUZReJbRtbMFiOVR0Gp_Q9OlkBbQtks5twdzagaJpZM4T2w9H
.

Seems like the maven compiler plugin allows configuration of main and test compiler arguments separately
http://maven.apache.org/plugins-archives/maven-compiler-plugin-2.5.1/compile-mojo.html#compilerArguments
http://maven.apache.org/plugins-archives/maven-compiler-plugin-2.5.1/testCompile-mojo.html#testCompilerArguments

Thanks @kageiit.  We actually need the compilerArgs parameter from more recent version to pass strings:
https://maven.apache.org/plugins/maven-compiler-plugin/compile-mojo.html#compilerArgs
https://maven.apache.org/plugins/maven-compiler-plugin/testCompile-mojo.html#compilerArgs
Our Maven example currently sets compilerArgs in the global configuration, which I think will apply to both the compile and testCompile phases.  Does someone know of an example somewhere that shows how to set compilerArgs differently for these two phases?  If someone gets me an example I can update the docs.

There actually seems to be no need to configure it for the two phases separately. I've had success just excluding the test sources path:
<compilerArgs>
  <arg>-Xep:NullAway:ERROR</arg>
  <arg>-XepOpt:NullAway:AnnotatedPackages=[...]</arg>
  <arg>-XepExcludedPaths:.*/src/test/java/.*</arg>
</compilerArgs>

@cbruegg yes that will work if you are using the default filesystem layout.  And it could be adapted if you're doing something else.  I still suspect there is some configuration magic for passing different compiler args for test vs. main code, but we are not Maven experts and haven't figured it out.


CUDA's reduce_by_key fails on sm_50 devices

Narrowed issue down to https://github.com/thrust/thrust/blob/master/thrust/system/cuda/detail/bulk/algorithm/reduce_by_key.hpp#L148
adding cassert before malloc fixes issue. keys_first and keys_last do not seem to be initialized properly so the for loop on line
https://github.com/thrust/thrust/blob/master/thrust/system/cuda/detail/bulk/algorithm/reduce_by_key.hpp#L159
is never executed.
#if __CUDA_ARCH__ >= 200
  cassert(keys_first < keys_last);
  size_type *s_flags = reinterpret_cast<size_type*>(bulk::malloc(g, interval_size * sizeof(int)));
  value_type *s_values = reinterpret_cast<value_type*>(bulk::malloc(g, interval_size * sizeof(value_type)));
#else

#include <thrust/device_vector.h>
#include <thrust/reduce.h>
#include <thrust/sequence.h>
#include <iostream>

int main(int argc, char ** argv)
{
    thrust::device_vector<int> indices(10);
    thrust::sequence(indices.begin(), indices.end());

    thrust::device_vector<int> temp(10, -1);

    thrust::counting_iterator<int> iter(0);
    thrust::reduce_by_key(thrust::make_zip_iterator(thrust::make_tuple(iter, iter)),
                          thrust::make_zip_iterator(thrust::make_tuple(iter, iter)) + temp.size(),
                          thrust::constant_iterator<int>(1),
                          thrust::make_discard_iterator(),
                          temp.begin(),
                          thrust::equal_to<thrust::tuple<int,int> >(),
                          thrust::plus<int>());

    std::copy(temp.begin(), temp.end(), std::ostream_iterator<int>(std::cout, " "));
    std::cout << std::endl;

    thrust::fill(temp.begin(), temp.end(), -1);

    thrust::reduce_by_key(thrust::make_zip_iterator(thrust::make_tuple(indices.begin(), indices.begin())),
                          thrust::make_zip_iterator(thrust::make_tuple(indices.end(), indices.end())),
                          thrust::constant_iterator<int>(1),
                          thrust::make_discard_iterator(),
                          temp.begin(),
                          thrust::equal_to<thrust::tuple<int,int> >(),
                          thrust::plus<int>());

    std::copy(temp.begin(), temp.end(), std::ostream_iterator<int>(std::cout, " "));
    std::cout << std::endl;

    return 0;
}

Nice work, Steve, thanks!
Are you saying the introduction of an assert() causes the issue to go away, or that your assert() seems to reveal the root cause of the issue?
I'm trying to figure out if Thrust is not initializing those iterators correctly, or if there is a compiler problem at root.

 If you agree, then we can probably close this bug.

assert() seems to make the issue go away. I think you are right about it being a code generation bug.
simply testing the condition and adding a empty printf fixes the error as well even though the code should never be executed.
closing this sounds good.
  #if __CUDA_ARCH__ >= 200
 #if __CUDA_ARCH__ >= 500
   if(keys_first > keys_last) printf("failed");
 #endif
    size_type *s_flags = reinterpret_cast<size_type*>(bulk::malloc(g, interval_size * sizeof(int)));
    value_type *s_values = reinterpret_cast<value_type*>(bulk::malloc(g, interval_size * sizeof(value_type)));
  #else
```:

Thanks Steve, nice work identifying the root cause. I'll close.

I believe it was a compiler bug, that was fixed recently.


Add step function

My attempt at rebasing the arrow and step functionality into our current dev HEAD
Still have some build errors and the functionality is a bit whack but I'm getting close, I wanted to make this branch accessible in case @daviroo or @0tuedon wanted to take a crack at it as well
Preview

I noticed we can't step or Run when we get our answer and we get a Success message when I just keep changing the stack

I noticed we can't step or Run when we get our answer and we get a Success message when I just keep changing the stack

Let's go over it today on the call.  I think we need to align on what the best behaviour is.

I noticed we can't step or Run when we get our answer and we get a Success message when I just keep changing the stack

Let's go over it today on the call. I think we need to align on what the best behaviour is.

Yh Looking forward to that


Parsing of generics within pattern match rules fails

This was opened on CodePlex by ranma42
From my understanding, all the three test{1,2,3} functions should be legitimate, but they result in the following errors:
F# Compiler for F# 3.1 (Open Source Edition)
Freely distributed under the Apache 2.0 Open Source License
/Users/ranma42/Code/sharp/datalog/test-match.fs(10,12): error FS0010: Unexpected identifier in pattern. Expected infix operator, quote symbol or other token.
/Users/ranma42/Code/sharp/datalog/test-match.fs(14,14): error FS0010: Unexpected type application  in pattern matching. Expected '->' or other token.
/Users/ranma42/Code/sharp/datalog/test-match.fs(18,16): error FS0010: Unexpected identifier in pattern. Expected infix operator, quote symbol or other token.
/Users/ranma42/Code/sharp/datalog/test-match.fs(18,8): error FS0583: Unmatched '('
I hit this issue while trying to pattern match on a provided type as in
let inline (|ParseRegex|_|) regex str =
  if (^T : (static member IsMatch : string -> bool) (str)) then
    Some (^T : (member Match : string -> 'U) (regex,str))
  else
    None
let parse str =
  match str with
    | ParseRegex RegexTyped<"test">.I m -> printfn "match %A" m
    | _ -> printfn "mismatch"

but I found out that I could not match in most (all?) cases where the rule includes a <...> syntax.
Is this a limitation of the parser?
file attachments
test-match.fs

comments
latkin wrote Jun 20, 2014 at 6:45 AM [x]
The example from the attached file captures the issue nicely (easier to understand than the regex example):
Basically, why does this work:

let (|Is|_|) v x =
  if v = x then
    Some x
  else
    None

>let intNone = Option<int>.None

>let test1 = function
  | Is intNone x -> "None"
  | _ -> "Not none"

but this doesn't

let test2 = function
  | Is (Option<int>.None) x -> "None" // invalid argument to parameterized pattern label
  | _ -> "Not none"

latkin wrote Jun 20, 2014 at 6:54 AM
Interesting, I get the above typecheck error in the IDE, but when I try to actually compile as source, >>I get parse error same as original report.
TestLibrary.fs(10,17): error FS0010: Unexpected identifier in pattern. Expected infix operator, quote symbol or other token.
TestLibrary.fs(10,8): error FS0583: Unmatched '('

This is expected - the F# language has never accepted type parameters in pattern position like this
Please add an entry at http://fslang.uservoice.com to suggest lifting this limitation?
Thanks!
Don


feat(CertManager): (WIP) Console Plugin Incept

Thanks for this, however as discussed last week this plugin is going to be more focused on just OpenShift GitOps. I plan on moving the External Secrets stuff in it to a new plugin in the near future. If you want to contribute this to a separate cert-manager plugin that would be awesome.


cairo: upstream fix for macOS Big Sur (or newer) quartz crash

Description of changes

This adds a patch for two commits from cairo upstream, fixing a crash on macOS Big Sur or
newer that occurs with the Quartz backend. The patch file containing the two
commits was copied from MacPorts.
For the original issue, see:
https://gitlab.freedesktop.org/cairo/cairo/-/issues/420
Things done
I manually tested this fix on a GTK 3 application I am trying to patch to get to run on aarch64-darwin (the application in question is xournalpp). I can confirm that the cairo patch fixes the crash that occurs on xournalpp.

Here is the flake I used to test with:
Run nix develop -c xournalpp
{
  inputs.flake-utils.url = "github:numtide/flake-utils/v1.0.0";

  outputs = { self, nixpkgs, flake-utils, xournalpp_src }: (
    {
      overlays = {
        default = final: prev: {
          cairo = prev.cairo.overrideAttrs (attrs: {
            patches = attrs.patches ++ [ ./patch-cairo-quartz-surfaces.diff ];
          });
          xournalpp = prev.xournalpp.overrideAttrs (attrs: {
            meta = {};
	    nativeBuildInputs = attrs.nativeBuildInputs ++ [ final.ninja ];
            cmakeFlags = (attrs.cmakeFlags or []) ++ ["-DCMAKE_DISABLE_FIND_PACKAGE_X11=on"];
          });
        };
      };
    } //
    (flake-utils.lib.eachDefaultSystem (system:
    let pkgs = import nixpkgs {
      inherit system;
      overlays = [ self.overlays.default ];
    };
    in
    {
      packages = flake-utils.lib.flattenTree {
        xournalpp = pkgs.xournalpp;
        cairo = pkgs.cairo;
        default = pkgs.xournalpp;
      };
      devShells = flake-utils.lib.flattenTree {
        default = pkgs.mkShell {
          nativeBuildInputs = with pkgs; [
            xournalpp hicolor-icon-theme gnome.adwaita-icon-theme
          ];
        };
      };
    }))
  );
}

Built on platform(s)

[ ] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[x] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
22.11 Release Notes (or backporting 22.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module
[ ] (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes

[ ] Fits CONTRIBUTING.md.

Can't get xournalpp to build on macos to test this.
/tmp/nix-build-xournalpp-1.1.1.drv-0/source/src/gui/inputdevices/touchdisable/TouchDisableX11.cpp:5:10: fatal error: 'gdk/gdkx.h' file not found

Any other applications exhibit the crash?

Oops, somehow I buried the flake that I tested with in the middle of the test checklist.

Here is the flake I used to test with
Run nix develop -c xournalpp
{
  inputs.flake-utils.url = "github:numtide/flake-utils/v1.0.0";

  outputs = { self, nixpkgs, flake-utils, xournalpp_src }: (
    {
      overlays = {
        default = final: prev: {
          cairo = prev.cairo.overrideAttrs (attrs: {
            patches = attrs.patches ++ [ ./patch-cairo-quartz-surfaces.diff ];
          });
          xournalpp = prev.xournalpp.overrideAttrs (attrs: {
            meta = {};
	    nativeBuildInputs = attrs.nativeBuildInputs ++ [ final.ninja ];
            cmakeFlags = (attrs.cmakeFlags or []) ++ ["-DCMAKE_DISABLE_FIND_PACKAGE_X11=on"];
          });
        };
      };
    } //
    (flake-utils.lib.eachDefaultSystem (system:
    let pkgs = import nixpkgs {
      inherit system;
      overlays = [ self.overlays.default ];
    };
    in
    {
      packages = flake-utils.lib.flattenTree {
        xournalpp = pkgs.xournalpp;
        cairo = pkgs.cairo;
        default = pkgs.xournalpp;
      };
      devShells = flake-utils.lib.flattenTree {
        default = pkgs.mkShell {
          nativeBuildInputs = with pkgs; [
            xournalpp hicolor-icon-theme gnome.adwaita-icon-theme
          ];
        };
      };
    }))
  );
}

You have to add -DCMAKE_DISABLE_FIND_PACKAGE_X11=on to the xournalpp cmakeFlags to get it to build.
Unfortunately, I was unable to build the obvious other candidate packages that I could think of (gimp and inkscape).

The upstream fix is added in #178551

I wasn't aware of #178551, and I can confirm it fixes the issue. Sorry for the confusion.

Update: this pr and https://github.com/NixOS/nixpkgs/pull/178551 have a small difference in the patch.

This patch from MacPorts also includes https://gitlab.freedesktop.org/cairo/cairo/-/merge_requests/119 , which I think also fixes a memory leak.

This patch from MacPorts also includes https://gitlab.freedesktop.org/cairo/cairo/-/merge_requests/119 , which I think also fixes a memory leak.

Sure, you can create a new pr for that fix.


fix: add @AirBnB logo

add @AirBnB logo

Why not just add all of the SimpleIcons? ;d

Icons in simple-icons doesn't have default color, and some are colored by fill attribute, some by stroke. Auto processing (add color attr) works not so well, so I removed it, decide every icon need to get reviewed before it's available.

Right.
Is airbnb's okey for merge?

This one need a fill="#FFF" too 😂
After svgomged:
<svg xmlns="http://www.w3.org/2000/svg" fill="#FFF" viewBox="0 0 24 24">
  <path d="M12 18.3a11.3 11.3 0 0 1-2.4-4.5c-.3-1-.2-1.8.3-2.5s1.2-1 2.1-1 1.6.3 2.1 1 .6 1.5.3 2.5c-.3 1.3-1 2.8-2.4 4.5zm9.6 1.1c-.2 1.3-1 2.3-2.2 2.8-2.3 1-4.5-.6-6.4-2.7 3.2-4 3.7-7 2.4-9-.8-1.2-2-1.7-3.4-1.7-3 0-4.6 2.5-4 5.4.4 1.5 1.4 3.3 3 5.3-1 1-2 1.8-2.7 2.3-.7.4-1.3.6-1.9.6-2.6.4-4.7-2.2-3.8-4.8l.9-2c1.4-3.2 3.2-6.8 5.3-10.9l.6-1.2c.4-.8.6-1.2 1.3-1.6.4-.3.8-.4 1.3-.4 1 0 1.7.6 2 1l.6 1 .5 1 .1.2 5.3 10.8.6 <IP_ADDRESS>c.2.7.3 1.3.2 1.9zm1.2-2.4l-.9-2v-.1c-1.9-4-3.6-7.6-5.3-10.8l-.1-.2C15.3 1.5 14.5 0 12 0 9.6 0 8.5 1.7 7.5 3.9l-.1.2C5.7 7.3 4 10.9 2 14.9l-.6 1.3-.3.8A5.1 5.1 0 0 0 6 24H6.6c1.8-.2 3.6-1.4 5.4-3.4 1.8 2 3.6 3.2 5.4 3.4h.6c3.4 0 6.2-3.3 4.8-7z"/>
</svg>

Aha. I'll update it.


Firefox headless browser

Is there any way for running on firefox headless mode?

Hopefully this is of some help. There wasn't a lot of guidance on how to do this with a json config, typically you see lots of code examples. What I ended up doing was using python to build the config I know works, then used the to_capabilities() method on the options object to get the JSON representation.
So, this Golem Selenium Grid Docker-Compose Example includes my config for remote devices with a headless chrome laptop-sized device, a headless chrome mobile-sized device, then a headless firefox device.
One thing to note is that I'm using docker selenium grid nodes, and I had to share the host machines memory with the nodes, otherwise the firefox node would randomly timeout and error out. That is configured in the docker-compose file.

@NihadMemmedli support for Firefox headless was added. Not yet released.
@rothnic That's awesome man, thanks for sharing.

@lucianopuccio I ended up packaging it up a little bit better in golem-docker. I haven't quite gotten to testing it out starting from a clean slate, but it is mostly there.


pipenv sync doesn't remove packages

If I remove a package from Pipfile.lock and perform pipenv sync, I expect that the removed packaged is removed from the virtual environment. This, however, doesn't happen. it's still present in .venv/lib/Python3.7/site-packages.

This is an API change, you may need to submit a PEEP to the peeps dir via a normal PR.

So, you mean pipenv sync is not supposed to sync the removed packages? :)
In this case, the workflow should always be pipenv clean plus pipenv sync in case you don't know if some packages have been removed or only added, right?

Right

Pipenv sync installs packages, it is not a destructive operation (if you want, it might be reasonable to propose a flag for this via a PEEP)
Closing for now, thanks for the report!


If model in session throws NotSerializableException - do not store in DynamoDB

I've just had a look at that DynamoSessionItem by (new String(sessionItem.sessionData.array())) in debugger. It shows that session data already contains model with stacktrace for NotSerializableException . I think such behavior should be changed.  I'm having trouble reading the dump above.

Sure, I can't imagine more exiting activity then reading dumps too :) Almost like in "The Matrix"!
Ok, now seriously, the issue is that my session had a not serializable object. Session manager tried to serialized it, of cause NotSerializableException poped up. At this point I'd expect session manager to complain to log file about this issue and ignore unserialazable object tree. But instead of ignoring, session manager managed to serialized NotSerializableException and all the long stack trace and put it into DynamoDB.
The good thing here is that among different garbage the dump contains kind of human readable names of classes, so by looking through it we can get a general idea about object types and their sequence how they are stored in this session. If we search for "NotSerializableException" and start to look after we'd see that now session contains full stack trace with references to DynamoDB, Spring, Tomcat etc - all the system internal, something that should never be in session.
The reason why I attached the dump was to prove that DynamoDB now contains something that it shouldn't.

Just imagine that one application has a bug where under very rare use case which wasn't captured by QA there is a not serializable object sneaks into session. What happens in production?:
Case 1)  when session manager would ignore such a session, user would see something like 500 error, would open a different page and it would work because session manager would load the latest valid session from DynamoDb. So user could continue doing other things.
Case 2) like in the bug. Session manager has persisted invalid session object. On the next request tomcat reads session from DynamoDB and it will never be deserialized again. Now user can't visit other pages, can't logout - nothing. Until session expires user can't continue working from the same browser. To make it worth, some sessions have very long expiration time because they are out of memory, some systems have functionality if you log in from a different http session it can check by UID and try to transfer existing session. Those user are doomed for a long time !
For safety I'd add two things:

don't persist session on "NotSerializableException"
if session can not be recovered - delete that object from DynamoDB (this should be configurable).

The way it works now it's a bit scary to use on production, as similar bug can prevent user from accessing entire web app.

Okay so here is what I found.
Standard session makes an attempt to deal with non-serializable attributes. writeObject (which we use to serialize the session data) skips any attributes which are outright not serializable. This is a very shallow check and only checks if the attribute object itself implements Serializable and doesn't assert on any other objects in the object graph. Any errors occurred during the actual serialization (using ObjectOutputStream) get handled a little differently. Here writeObject catches the NonSerializableException and writes a special String ("NOT_SERIALIZABLE_EXCEPTION") to the output stream to indicate the attribute was not able to be serialized. Now that would be fine as readObject has logic to ignore deserializing any attribute with that special string but unfortunately ObjectOutputStream and ObjectInputStream swoop in to ruin the day. When ObjectOutputStream encounters a non-serializable object it to writes a special indicator (TC_EXCEPTION) and the serialized exception object (which includes the stack trace). This is written internally before "NOT_SERIALIZABLE_EXCEPTION". Now when StandardSession.readObject goes to load the session data it uses ObjectInputStream which is explicitly checking for the TC_EXCEPTION indicator and if it is encountered it throws a WriteAbortedException wrapping the deserialized exception object. Because it fails fast the logic readObject has for skipping attributes that were serialized as "NOT_SERIALIZABLE_EXCEPTION" never gets executed and the session deserialization dies.
So for steps forward and to address your suggestions

I don't think we will be able to reliably detect if a session was persisted with non-serializable objects in the graph. Exceptions are only thrown when attempting to read the written data. We theoretically could write it then attempt to immediately read the binary data back into a session but this would be pretty inefficient.
I agree we should have some way for the session manager to purge invalid or corrupt sessions with some error/fatal logging for good measure.

Thanks for the explanations.

Yep, I see, it's org.apache.catalina.session.StandardSession fault and we can't help it.
However StandardSession serialization idea doesn't look safe. Let's try business/dao layers analogy. Usually DAO layer should return exactly what it saved. It shouldn't change the domain/business model it persists. It is pure responsibility of business layer. In this case DAO is tomcat session manager or StandardSession,writeObject method, and service layer or client is MVC controller. If I pass an object to DTO I expect to either receive exactly the same copy later or a notification that something bad has happened. The last thing I want is silent interference of DTO on my business model.
That would be nice to have.

Thank you!

I 100% agree. Everytime I have to dig into the code for StandardSession I die a little inside. In version 1.0 we actually had a custom serialization scheme but due to some strange side affects in the public interface of StandardSession it caused several bugs. See https://github.com/aws/aws-dynamodb-session-tomcat/pull/19 if you're interested in the backstory for this. I'll ruminate on the issue and see if there is anything we can do to fail faster in this specific scenario.
I'll get this and the changes to remove the Converter chain coded up and prepped for release.

Opened https://bz.apache.org/bugzilla/show_bug.cgi?id=58284 to see if it can be fixed internally before trying to hack around it.

Got an update on the bug report. Proposed solution is to fix what the code is intending to do and skip non-serializable attributes on write and read and log an error. Considering the alternative (throwing exceptions and failing fast) is a breaking change their proposition makes sense. Not ideal as it could hide serious bugs in the users code but it's a little better then writing successfully and dieing on the read. I'll keep this issue open and updated as the bug fix progresses.

Just wanted to let you know I haven't forgotten about this. Was out of the office last week and picking this back up. I apologize for the delay.

This has been released in https://github.com/aws/aws-dynamodb-session-tomcat/releases/tag/v2.0.1. Also https://bz.apache.org/bugzilla/show_bug.cgi?id=58284 was accepted and is available in Tomcat 7.0.65 and 8.0.27.

Excellent news! Thank you, Andrew!


Admin vs Non-Admin Usage & Settings UX Update

This PR is a large chunk of the requirements from #3313
It is split off from #3647 to avoid issues with Salesforce Integration.

@ShadeWyrm can you please add some test instructions to the PR description so folks without context can review without too much digging on what the PR does ?

@ShadeWyrm this is looking good
Thinking we can remove the getter / setter for the useTemplateStore  i.e.  https://github.com/cds-snc/platform-forms-client/commit/3af5a8e96726b3eaa0c18989a57300be6aeedcdb does that work?

@ShadeWyrm this is looking good
Thinking we can remove the getter / setter for the useTemplateStore i.e. 3af5a8e does that work?
Get
 formPurpose: s.formPurpose,

Update
 updateField("formPurpose", purposeOption);

Yah that should work just fine, it was a hold over from my testing, and figuring out the template store. I had assumed it was necessary, but the Linter brought that into question on my end. :) So just the property should be enough.

https://github.com/cds-snc/platform-forms-client/pull/3714#issuecomment-2142090035
Do you want to cherry pick that commit in or make the updates on you end?


Empty macros don't work (anymore?)

I'm pretty sure this worked at some point:
#! MYMACRO=

Hmm, let me check when the last commit that worked in was...

Actually, I lied. This apparently never worked. Oops...

Should we allow it?

Empty macros actually worked for me but there's no downside to the slight change in regex


Fix cpp compatibility

When using CCL in external c++ code, variable length arrays in function declaration / definitions were throwing compile errors. Only in ccl_background.h and ccl_background.c files.

Maybe also add __cplusplus guards into all header files?


NTR: [purine nucleotide uniporter activity]

Please provide as much information as you can:

Suggested term label:
purine nucleotide uniporter activity

Definition (free text)
Catalysis of the active transport of purine nucleotides across a membrane by a mechanism involving conformational change, where energy for active transport is derived from membrane potential if the solute is charged.

Reference, in format PMID:#######
(REQUIRED)
PubMed:18375752, PubMed:27477609

Gene product name and ID to be annotated to this term
Slc17a9

Parent term(s)
GO:0022810  membrane potential driven uniporter activity
GO:0015216  purine nucleotide transmembrane transporter activity
GO:0015292  uniporter activity

Children terms (if applicable) Should any existing terms that should be moved underneath this new proposed term?

Synonyms (please specify, EXACT, BROAD, NARROW or RELATED)

Cross-references

For enzymes, please provide RHEA and/or EC numbers.

Can also provide MetaCyc, KEGG, Wikipedia, and other links.

Any other information

+[Term]
+id: GO:0160042
+name: purine nucleotide uniporter activity
+namespace: molecular_function
+def: "Catalysis of the active transport of purine nucleotides across a membrane by a mechanism involving conformational change, where energy for active transport is derived from membrane potential if the solute is charged." [PMID:18375752, PMID:27477609]
+intersection_of: GO:0022810 ! membrane potential driven uniporter activity
+intersection_of: has_primary_input CHEBI:26395 ! purine nucleotide
+created_by: rynl
+creation_date: 2023-03-11T00:58:42Z

Done


Added a simple code example for DataTable

Good examples, thanks you. I just added a small change in the wording (replacing "I" with "we").


Add Colored Output

I think it would be cool to use something like this to color the X's and O's on the board

@2016rshah I've never programmed Haskell. I'll try it!


spatial Shader working in 4.2.2 but not 4.3

Tested versions
Reproducible in v4.3 but not in v4.2.2
System information
Windows 11 - Godot v4.3 - Vulkan (Forward+) - dedicated
Issue description
I am using a shader based on StayAtHomeDev water shader: https://godotshaders.com/shader/water-shader/
This is applied to a MeshInstance3D with a QuadMesh on it.
In Godot 4.2.2 this shader is working to create water, but in 4.3there are some issues. There is no error in the console, only visual errors you apply the shader.
Steps to reproduce
Just view the scene in the editor in 4.3 and 4.2.2
Minimal reproduction project (MRP)
https://github.com/CaelRowley/godot-shader-error

Shader code should be changed due to reverse z (https://godotengine.org/article/introducing-reverse-z/)

@arkology ok thanks for the link, I was searching the change logs to see if they altered shaders and couldn't find anything and I didn't get any warning about the breaking change like the article describes, but I should be able to fix it now, thanks!


feat: split backend and frontend service

Description
This PR is a part of a project to split up the Python WAAS API and the Frontend into two separate services
The end goal is to be able to deploy the Astro frontend and Python backend on their own.
Also choosing a more suitable frontend framework (in this case we chose Astro) should make it easier to contribute to the frontend codebase
Bonus goals:

Split out (Pulse) tracking to its own modular module

Long term this will also make it easier for us to switch out the backend service with something else. Whisper c++, WhisperX, etc...

Effort moved to https://github.com/schibsted/WAAS/pull/240


Integrate hoodmaps?

While the content is kinda trash, it might be beneficial to add automatically generated links to hoodmaps https://hoodmaps.com/vilnius-neighborhood-map

I'm considering making separate pages for notable cities but maybe this can be made into a shortcode for country pages that have city-specific sections


RSpec/VariableName   gives" has the wrong namespace - should be Naming" when run with --only option

rubocop will run the cop unless you use the --only option
rubocop -f o
will give this output

9    RSpec/VariableName

Indicating 9 viiolations  found, but running

rubocop --only RSpec/VariableName

will give

--only option: RSpec/VariableName has the wrong namespace - should be Naming
Inspecting 28 files
............................
28 files inspected, no offenses detected

version of rubocop-rspec is 1.44.1
version of rubocop is 0.93.1
running in docker container based on Ruby 2.7.2 image from dockerhub

I hope we've fixed it with the soon-to-be-released 2.0. Can you please help with testing it? Point your Gemfile to release-2.0:
gem 'rubocop-rspec', github: 'rubocop-hq/rubocop-rspec', branch: 'release-2.0'

2.0 has been released, and this issue should be resolved.
Please feel free to reopen if the issue remains.


[Reputation Oracle] Completed webhook

Description
Send a webhook from Reputation Oracle to Job Launcher and Recording Oracle once the escrow is completed
*(Waiting #1620 to complete Job Launcher webhook handling)
Summary of changes
Add a new method inside webhook.service.ts to send a webhook
In cron-job.service.ts get the job launcher and recording oracle webhhook urls and send the webhook
Update all tests
How test the changes
yarn test
Related issues
#1625

https://github.com/humanprotocol/human-protocol/blob/74ff5cb7ed01333580a3bd840d699438c189898d/packages/apps/reputation-oracle/server/src/common/enums/role.ts#L2

Check JobLauncher typo


gnomeExtensions: update

Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
24.11 Release Notes (or backporting 23.11 and 24.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

@ofborg test gnome-extensions

No idea how to review such bumps properly but I've been using this PR since I upgraded to GNOME 47 and extensions that I usually use work properly


Create markdown-link-checker.yml

Trying to find a good working workflow that detects problem such as the ones being fixed in https://github.com/dotnet/docs/pull/20137.

I can't get either of the two workflows I found to work.
@adegeo Any ideas?

Hrm, not sure? But I dont think this would work anyhow. Does the tool check xref links? Those can be put inside of markdown links: [some link text](xref:api-path)
The build system at least tells us of any non HTTP link AFAIK.

I think xrefs shouldn't be detected as relative link. These actions should only work in relative links.

The way I got the initial snippets testing working to work was doing it in like your fork first. So I think the workflows only trigger when you're working on the branch they are in. So create branch1 with the workflow configured and ready, then do things in branch2 which has a PR into branch1 where the workflow exists. And I think your trigger needs to be on: pull_request not on: push as you want to test the links of the files when a bad PR comes in, not after it's merged.

@adegeo The work flows indeed was triggered in my fork (they doesn't appear in PR but I can see in my fork). However, they doesn't seem to be doing any good.
One of the workflows (https://github.com/Youssef1313/docs/actions?query=workflow%3A"Markdown+link+checker") passes  in 5 minutes, although they should report broken links.
The second one is actually very funny (https://github.com/Youssef1313/docs/actions?query=workflow%3A"Check+Markdown+links"), it fails after 6 hours with a timeout 😄

No idea really.. does the workflow work with PRs checking just the files in it? I would look into how to configure that script. A lot of scripts allow you set them up through properties or env variables.

@adegeo I'll try to play with it in my fork and see if I could get something to work. In the meanwhile, I'll close the PR and re-open if I could get to something working.


(Compose v2) Cant add own content - 502 Bad content // 111 Connection refused

OK, So I have cloned the example image and substituted the "example.com" domain with my own but when trying to add my own content, The browser gives me 502 Bad Gateway.
This is my config:
/examples/docker-compose/v2/simple-site/docker-compose.yml
version: "2"
services:
  nginx:
    image: nginx
    container_name: nginx
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - "/etc/nginx/conf.d"
      - "/etc/nginx/vhost.d"
      - "/usr/share/nginx/html"
      - "../../../volumes/proxy/certs:/etc/nginx/certs:ro"
    networks:
      - proxy-tier

  nginx-gen:
    image: jwilder/docker-gen
    container_name: nginx-gen
    volumes:
      - "/var/run/docker.sock:/tmp/docker.sock:ro"
      - "../../../volumes/proxy/templates/nginx-compose-v2.tmpl:/etc/docker-gen/templates/nginx.tmpl:ro"
    volumes_from:
      - nginx
    entrypoint: /usr/local/bin/docker-gen -notify-sighup nginx -watch -only-exposed -wait 5s:30s /etc/docker-gen/templates/nginx.tmpl /etc/nginx/conf.d/default.conf

  letsencrypt-nginx-proxy-companion:
    image: jrcs/letsencrypt-nginx-proxy-companion
    container_name: letsencrypt-nginx-proxy-companion
    volumes_from:
      - nginx
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock:ro"
      - "../../../volumes/proxy/certs:/etc/nginx/certs:rw"
    environment:
      - NGINX_DOCKER_GEN_CONTAINER=nginx-gen
      - DEBUG=true

  simple-site:
    image: nginx
    container_name: simple-site
    volumes:
      - "../../../volumes/examples/simple-site/conf.d/:/etc/nginx/conf.d"
      - "../../../volumes/examples/simple-site/site:/var/www/html"
    environment:
      - VIRTUAL_HOST=test.mydomain.com
      - VIRTUAL_NETWORK=nginx-proxy
      - VIRTUAL_PORT=80
      - LETSENCRYPT_HOST=test.mydomain.com
      -<EMAIL_ADDRESS>    networks:
      - proxy-tier

/examples/volumes/examples/simple-site/conf.d/test.mydomain.com
  server {
    listen                  80;
    server_name             test.mydomain.com;

    index index.html index.htm;

    root /var/www/html;

    location / {
                try_files index.htm index.html;
    }

}

/examples/volumes/examples/simple-site/site/index.html

<!DOCTYPE html>

ok

When running:

docker network create -d bridge nginx-proxy; docker-compose up -d

everything downloads and installs fine.
I had a similar problem.
I upgraded docker-compose with
'sudo pip install docker-compose --upgrade'
Now that I'm using docker-compose version 1.8.1, this works for me now.


single escape for writing SQL execution

self._conn.execute (line 296 of chromadb/db/duckdb.py) will throw a Binder Error if ids given to Clickhouse._create_where_clause (line 136 of chromadb/db/clickhouse.py) contains single quote. And the solution is to escape single quotes by replacing them to double single quotes (see https://stackoverflow.com/questions/1586560/how-do-i-escape-a-single-quote-in-sql-server). Meanwhile, single quotes will cause the tuple formatted with double quotes, which will cause other problem (see https://stackoverflow.com/questions/75671499/duckdb-binder-error-referenced-column-not-found-in-from-clause), therefore a two-step-replacement is performed.

@xinhjBrant thanks for this! can you post a small example of this breaking?

@xinhjBrant thanks for opening this! We are dropping duckdb and clickhouse tomorrow which should fix this up entirely. Appreciate it!


Build release infrastructure

Would like to do a release of Lakeroad well before the ASPLOS talk. Need to build the infrastructure for it.

[ ] Implement releasing for Ubuntu
[ ] Implement releasing for MacOS
[ ] Implement releasing for Windows
[ ] Put everything in a folder before it gets zipped -- currently just zipping binaries directly
[ ] Figure out how to package solvers with the binaries

[ ] Determine a clean way to provide default hard-coded paths to solver binaries in Lakeroad, ideally while letting users override

[ ] Package testing tools (or at least an example) in the release, so users can run tests
[ ] Similar to default hard-coded paths for binaries, figure out how to provide default hard-coded path from portfolio solver binary to single solver binary, ideally allowing for override
[x] #419
[x] #421
[ ] Copy solver binaries into Rosette installation location (needed during raco distribute)
[ ] Put binaries into Rosette

[x] #424

We have to transition over to building binaries statically, so that they can be fully packaged into Rosette. If we're going to do that, we might as well do it in Docker build too, so that things are consistent, which means moving away from oss-cad-suite.

Packaging Binaries
This comment pertains to how we should package up the various binaries needed by Lakeroad.
Some questions:

Licensing issues?
How do we ensure that the Lakeroad binaries utilize the packaged solvers by default?

Seems like I'm nearly there on Ubuntu, but getting this issue when testing the built version in an Ubuntu Docker image:
root@4420df0a3e22:~# ./lakeroad/bin/lakeroad --cvc5 --verilog-module-filepath one_stage_add_mul_add_signed_18_bit_xilinx.sv    --architecture xilinx-ultrascale-plus   --template dsp   --out-format verilog   --top-module-name top   --verilog-module-out-signal out:18   --initiation-interval 1   --clock-name clk   --module-name test_module   --input-signal a:18   --input-signal b:18   --input-signal c:18   --input-signal d:18   --extra-cycles 1   --timeout 120
error writing to stream port
  system error: Broken pipe; errno=32
  context...:
   .../smt/env.rkt:78:0: ref-expr!
   .../match/compiler.rkt:559:40: f161
   .../smt/env.rkt:78:0: ref-expr!
   .../match/compiler.rkt:559:40: f135
   .../smt/env.rkt:78:0: ref-expr!
   .../match/compiler.rkt:559:40: f28
   .../smt/env.rkt:78:0: ref-expr!
   .../match/compiler.rkt:559:40: f161
   .../smt/env.rkt:78:0: ref-expr!
   .../match/compiler.rkt:559:40: f135
   .../smt/env.rkt:78:0: ref-expr!
   .../match/compiler.rkt:559:40: f28
   .../smt/env.rkt:78:0: ref-expr!
   .../match/compiler.rkt:559:40: f161
   .../smt/env.rkt:78:0: ref-expr!
   .../match/compiler.rkt:559:40: f135
   ...

Interesting -- it works with --bitwuzla. So there's something up with the solver specifically. Doesn't work with cvc5, yices, or stp.

(putting more context at the start here for @sorawee)
I am doing a release of Lakeroad. Seems like, during raco distribute, Rosette expects to have the solver binaries packed inside of it. That's pretty convenient, as I didn't previously know how I was going to handle packaging up and shipping the solvers. It's convenient that Rosette handles it.
However, after building the binaries, moving them into Rosette, and doing raco distribute, most of the solvers don't work. Only Boolector and Bitwuzla do. When I run the binaries directly, the error becomes more clear:
If I try to run the cvc5 binary directly, I get a missing lib error:
root@4420df0a3e22:~# ./lakeroad/deps/lakeroad/lib/plt/lakeroad-single-solver/exts/ert/r1/bin/cvc5 --help

./lakeroad/deps/lakeroad/lib/plt/lakeroad-single-solver/exts/ert/r1/bin/cvc5: line 6: /root/lakeroad/deps/lakeroad/lib/plt/lakeroad-single-solver/exts/ert/r1/lib/ld-linux-x86-64.so.2: No such file or directory

Same thing for cvc4, yices. For STP, it complains about missing libstp.so.2.3. Boolector and Bitwuzla work.
Maybe the answer here is to build the solvers statically? @sorawee , what do you think?

Main problem I'm running into right now is that I have to build the solvers and package them, but

I have to package them with their shared libraries, which means I need to change their RUNPATHs, which feels hacky/wrong, or
I have to build them statically, which is apparently a bad idea?

It's weird that building relocatable binaries isn't apparently more the norm.
Here's a helpful post: https://nehckl0.medium.com/creating-relocatable-linux-executables-by-setting-rpath-with-origin-45de573a2e98

One option is to set LD_LIBRARY_PATH in a wrapper script. That works for CVC5. Seems to work a bit for STP but STP has a new problem of missing a library installed by libboost-program-options-dev.

at a certain point, it's also going to be easier to give up on specific solvers (STP seems like a likely candidate)

I'm mostly there with the solvers now. Now the problem I'm having is this: the solver binaries copied into Rosette should be symbolic links. I'm 99% sure I am making them symbolic links. Yup, just checked, they are. But then, I think when we do raco distribute, they get copied over as binaries and not symbolic links. The problem here is that then those binaries too need their RPATHs set, if they're gonna be binaries. better to be symbolic links to the binaries we've already modified. Some options:

Just copy over the binaries once we do raco distribute
Set the rpaths on those binaries

Issue here: https://github.com/racket/racket/issues/4945

Added a hacky fix that just re-links.

Ok, next problem is that the symbolic links need to be relative: https://superuser.com/questions/146231/how-do-i-create-a-relative-symbolic-link-in-linux


merge MONDO:0006788 hydrophthalmos to MONDO:0020366 congenital glaucoma

Closes #7282

I'm looking into the qc errors

@sabrinatoro I changed the Orphanet to equivalentTo as you suggested. However that now causes a proxy merge so I guess that's why it was not there before and I am removing it.

@twhetzel, I am sure you did, but I am still asking: did you confirm that the Orphanet x-ref should be on the other term and not this one?


Python Function 1: string parser

Needs:

[x] functionality
[x] unit tests

Finish functionality with 100% coverage unit tests.


update logging info for solutions list

Description
I previously updated the logger info so that when calling solve for a list of inputs it didn't print the termination reason (since it may be different for each set of inputs). At the same time I changed it so that it returned the sum of the integration, solve and total times. This was the wrong thing to do as it gives the impression that the call to solve to ~N times as long as it actually did (for a list on N inputs). I've reverted it back to just print a single set up, solve and total time (note all solutions get given the same solve time, see https://github.com/pybamm-team/PyBaMM/pull/1261). I also removed the integration time info from the logger in this case since it is also different for each set of inputs. Users can still access the integration time for each solution in the list of solutions if they want to.
The logger info when returning a single solution remains unchanged.

Yeah I did think about that, but thought it might clutter up the info. I'll leave it for now and people can inspect the integration times for each solution manually. Maybe we can revisit this though as people start using this functionality more.

Yeah I did think about that, but thought it might clutter up the info. I'll leave it for now and people can inspect the integration times for each solution manually. Maybe we can revisit this though as people start using this functionality more.


Update/fix the Dockerfiles to use multi-stage builds to reduce container image sizes

[ ] I have added the tests to cover my changes.
[x] I have updated the documentation and CHANGELOG accordingly.
[x] I have read the CONTRIBUTING document.

pre-commit.ci autofix


Add token-based authentication to API

In support of mobile app.
First draft, as discussed in general email, is simply to allow app to pass in the user's password when creating an API key.  If the password is correct, it verifies the API key immediately, and it functions -- so far as I can tell -- identically to a normal token.
We can discuss additional features, such as having API keys automatically expire after a certain amount of time.  And whether we want certain apps to be considered "trusted" and exactly what that means.  (At the moment, I can't think of anything a "trusted" app would be able to do that I'm not comfortable giving every app the ability to do!)
Consider this the opening of a dialog about implementation details...

Coverage increased (+0.03%) to 93.104% when pulling e54c349feba4f8b3745d88478327879a47cae48c on pellaea:api-tokens into 818ca148ebe6325a6184b1a6580c2a0107badec9 on MushroomObserver:master.

Awesome!  I'll go ahead and merge it, then.
On Sat, Sep 18, 2021 at 1:34 PM Olivia Computer @.***>
wrote:

@.**** commented on this pull request.
Just tried this after resetting my dev environment. It is working for me
and should fit what we need! Thanks so much
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/MushroomObserver/mushroom-observer/pull/790#pullrequestreview-758079597,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAYTNNI5WRIWN5IVG5WNWCLUCTESFANCNFSM5EH5WNRQ
.
Triage notifications on the go with GitHub Mobile for iOS
https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675
or Android
https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.


No default namespace for resource

When I attempt to create a Job without Namespace: k8s.String("default") specified in the Metadata I get a no resource namespace provided error. Is this the expected behavior?

No. Can you copy your full Job constructor? This has tests for other resources:
https://github.com/ericchiang/k8s/blob/677cf3318ef83bf681a38821f81a233a9be09641/client_test.go#L170

I think I wasn't clear. I mean I get the error if I try to create a Job without a Namespace specified, expecting it to default to the "default" namespace. My job constructor (error if I remove the Namespace field):
  k8sClient, err = k8s.NewInClusterClient()
  if err != nil {
    log.Fatal(err)
  }

  job := &batchv1.Job{
    Metadata: &metav1.ObjectMeta{
      GenerateName: k8s.String(name + "-update-"),
      Namespace:    k8s.String("default"),
    },
    Spec: &batchv1.JobSpec{
      Template: &corev1.PodTemplateSpec{
        Spec: &corev1.PodSpec{
          Containers: []*corev1.Container{
            {
              Name:  k8s.String("main"),
              Image: k8s.String(image),
              EnvFrom: []*corev1.EnvFromSource{
                {
                  SecretRef: &corev1.SecretEnvSource{
                    LocalObjectReference: &corev1.LocalObjectReference{
                      Name: k8s.String("testdb"),
                    },
                  },
                  Prefix: k8s.String("DB"),
                },
              },
              VolumeMounts: []*corev1.VolumeMount{
                {
                  Name:      k8s.String("config"),
                  MountPath: k8s.String("/config"),
                },
              },
            },
          },
          RestartPolicy: k8s.String("Never"),
          Volumes: []*corev1.Volume{
            {
              Name: k8s.String("config"),
              VolumeSource: &corev1.VolumeSource{
                ConfigMap: &corev1.ConfigMapVolumeSource{
                  LocalObjectReference: &corev1.LocalObjectReference{
                    Name: k8s.String(name),
                  },
                },
              },
            },
          },
        },
      },
    },
  }
  if err := k8sClient.Create(ctx, job); err != nil {
    log.Fatal(err)
  }
  fmt.Printf("Job created: %q\n", *job.Metadata.Name)

At the least I'd expect it to default to Client.Namespace. If that's not specified falling back to "default" seems odd, though I guess that's what kubectl does?
Regards being explicit feels better than being implicit. Is it tedious to specify a namespace? I could imagine it making code harder to read by obfuscating which objects are namespaced or what namespace is being defaulted to.

Yes, kubectl assumes "default" namespace unless a different namespace is set in the context. So the analogous behavior would be setting Client.Namespace to "default" unless otherwise specified.

On a closer look, no namespace preference is set in the context unless specified manually, so the behavior is as you noted, with commands falling back to "default".

On closer inspection the comment for client.Namespace is:
    // Default namespaces for objects that don't supply a namespace in
    // their object metadata.
    Namespace string

Which I don't think actually gets applied anywhere. That does default to "default" if you don't supply one in your kubeconfig.
I think the fix would be to propagate that value correctly on a create and other actions.
It's going to be a second before I can write up a fix. Please feel free to send one if you'd like to.

Tried to do this today, but there are some subtitles in the API that I don't think justify making "" mean "use the default namespace"
For now users can continue to supply the namespace as a value:
client, err := k8s.NewClient(config)
if err != nil {
	// handle error
}
cm := v1.ConfigMap{
	Metadata: &metav1.ObjectMeta{
		Name:      &k8s.String("my-configmap"),
		Namespace: &client.Namespace,
	},
	Data: map[string]string{"foo": "bar", "spam": "eggs"},
}
err := client.Create(ctx, cm)

Some more thoughts in https://github.com/ericchiang/k8s/pull/97
For now, I'm going to go with a docs only "fix"
If we want to something more clever later, we still can. For now going to close this.


Add E-Labyrinth (Rolling Sky Soundtrack) by Cheetah Mobile

Song Name
E-Labyrinth (Rolling Sky Soundtrack)
Artist Name
Cheetah Mobile
Source
Youtube
Youtube Link
p4QxNPMnCKI
Direct File Link
No response
Song ID
746536
Start Offset [ms]
0
Extra
Created from Auto Nong.
Created for level: 63337478

accept


Replace the deprecated serde_yaml dependency

Request
The serde_yaml dependency has been deprecated upstream
Proposed solution
One possible alternative is the serde_yml crate, which I believe is pretty much a drop-in replacement.
Anything else?
No response

Polar

Thanks for letting me now. There's no rush on this so let's see how thing develop with serde_yml.

Thanks for letting me now. There's no rush on this so let's see how thing develop with serde_yml.

Hmm... would you be open for a submission that migrates to the non deprecated one or not? I'm happy to send a PR if that's possible.
(My motivation is your crate is a dependency of my own crate clap_allgen and deprecated deps can have issues as time progresses)
Thanks for your time! 👋

Thanks for letting me now. There's no rush on this so let's see how thing develop with serde_yml.

Thank you for the update!
Switching to a maintained alternative to serde_yaml sooner rather than later would be helpful for me:
Unfortunately, the serde_yaml dependency makes projects that use carapace-sh harder to package for distributions like Debian (this is what prompted me to open this ticket).
But then of course software engineering-life is full of tradeoffs and mutually contradictory goals, so I understand if you prefer to move more slowly.

Ah, ok then. I don't mind switching.
serde_yml does look to be the most promising fork, so let's go with it.

So serde_yml adds unnecessary quotes to map keys:
-  -V, --version: Print version
+  '-V, --version': Print version

Which is something i'd like to avoid.
And there some aspects to it that don't make me put my trust in it (for now).
Issue tracker also isn't active on the project so it's not clear if the project is really meant for public use.
serde_yaml_ng on the other hand at least announced to be open for external contributions.
It's unclear where things go with yaml support in rust which is why i was planning to wait a little.
I'm fine with using serde_yaml_ng for now, but be aware that all we do is getting rid of a "deprecated" notice on a project which is otherwise completely fine.

Understood (and your perspectives make sense to me).
Thank you for doing this work already now, even though the medium term future of yaml in Rust looks unclear indeed!


Error when running optimized code with SEAL: encrypted_ntt and plain_ntt parameter mismatch

I am getting the following error when running the optimized code for ResNet with SEAL:
terminate called after throwing an instance of 'std::invalid_argument'   what():  encrypted_ntt and plain_ntt parameter mismatch
Below are the commands I ran:

hc-trace ResNet
hbt dacapo 40 ResNet
hc-test dacapo 40 ResNet SEAL CPU

How can I resolve this issue?

Currently, SEAL does not support bootstrapping, so it cannot be tested as such. As a temporary solution, we have implemented SEAL's Encryption and Decryption to function similarly to bootstrapping. Please refer to the Tutorial-Test Code for details. However, this method cannot be considered true bootstrapping and is not recommended.
Due to the current HEAAN license issue, we are unable to test the results of the DaCapo paper. Additional research is ongoing in this area. Thank you for your interest!


Create schema  from a json object

Hey can the library support creating the schema from a json object
Example
 const JSONEXAMPLE = {
    "checked": false,
    "dimensions": {
        "width": 5,
        "height": 10
    },
    "id": 1,
    "name": "A green door",
    "price": 12.5,
    "tags": [
        "home",
        "green"
    ]
}

and some function like ajv.createSchema(JSONEXAMPLE)
if there is any please point towards the right direction other wise can you add this feature ?

Hi - it’s definitely out of scope of the library I’m afraid… I think there must be some other library that does it though, I can’t remember off the top of my head.
In general, you cannot derive a schema from data, but you can create some template of course - if that’s what you need it’s just a loop traversing the object and constructing the schema as it goes - it could be in the application code wherever you need it.


Fix pytest baselines path for PAX MGMN metrics

I must have forgotten to correct this when I restructured the files.

Good find. This change made me realize we probably do need a step in the rosetta tests that fails b/c as of now the tests look like they all passed, but the PAX perf tests failed, but that's buried in the jobs that were run.
I made an issue to track this, but that is not blocking this change: https://github.com/NVIDIA/JAX-Toolbox/issues/205


New version: MPC-BE.MPC-BE version 1.8.2

Created by 🥟 Dumplings in workflow run #10090.

Log
Updated: 1.8.1 → 1.8.2
Submitting WinGet manifests
Creating manifests
Uploading manifests and making commits
Creating a pull request

Microsoft Reviewers: Open in CodeFlow

@wingetbot run


feat: More from v2

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+0.2%) to 93.399%

Totals

Change from base Build<PHONE_NUMBER>:
0.2%

Covered Lines:
9112

Relevant Lines:
9756

💛  - Coveralls


[Task] implement DELETE /api/workspaces/{workspace_id}

Story: #458
Description
See story

Blocked waiting on WorkspaceServiceRepository to be implemented in the POST @LizaShak is working on - will pick this up once her PR goes through


Archive arm dartsdks on every revision

Currently the dartsdk builders do not seem to archive arm sdk zip files.
The dartsdk bundles can be found e.g. here for a bleeding edge revision and it is missing an arm version:
$ gsutil ls gs://dart-archive/channels/be/raw/136190/sdk/
...

Same is true for all builders, also for stable (see gs://dart-archive/channels/stable/raw/1.14.1/sdk). There is also no support for that in the GCSNamer.
But they suddenly appear in the promoted release folder. Comparing the timestamps on them (e.g. gsutil ls -lh gs://dart-archive/channels/stable/release/1.14.1/sdk) it looks like they are uploaded from the side from somewhere else. From where?
@whesse Can't we have dartsdks for arm be archived as everything else on each revision?
Some context: Dartino is using DartVM binaries from specific revisions (sometimes from bleeding edge) and we have to get the binary for linux/linux-arm/mac/windows. So for now we need to log into the buildbot to get them, since they don't seem to be automatically archived.
cc @ricowind

Yes, this is clear, we need to create the SDK on the builders rather than creating one afterwards by a hack.

Fixed.


Issue 2062: Added custom header precedence info

Let me know if anything should be adjusted.

\o/ This is beautiful @benjaminran! Thanks so much! :cake: :sparkles:

This is a fantastic pull request. Thanks so much!


Marathon is unsure when task was started

Notice that between two invocations startedAt changed:

2014-12-27T16:47:36.852Z
2014-12-27T16:48:06.907Z

Everything else is not changed.
~ λ curl -s http://marathon:8080/v2/tasks | jq .tasks[0]
{
  "appId": "/topface/collectd/es/analytics/web487",
  "id": "topface_collectd_es_analytics_web487.0b9cc007-8d5b-11e4-9c4e-56847afe9799",
  "host": "web566",
  "ports": [
    31313
  ],
  "startedAt": "2014-12-27T16:47:36.852Z",
  "stagedAt": "2014-12-26T23:58:10.432Z",
  "version": "2014-12-03T17:36:32.503Z",
  "servicePorts": [
    10008
  ]
}
~ λ curl -s http://marathon:8080/v2/tasks | jq .tasks[0]
{
  "appId": "/topface/collectd/es/analytics/web487",
  "id": "topface_collectd_es_analytics_web487.0b9cc007-8d5b-11e4-9c4e-56847afe9799",
  "host": "web566",
  "ports": [
    31313
  ],
  "startedAt": "2014-12-27T16:48:06.907Z",
  "stagedAt": "2014-12-26T23:58:10.432Z",
  "version": "2014-12-03T17:36:32.503Z",
  "servicePorts": [
    10008
  ]
}

Diff for the whole fleet of services running on marathon shows the same picture: startedAt changes from time to time.

The startedAt field should only be set upon receiving a TASK_RUNNING for the first time.  See: https://github.com/mesosphere/marathon/blob/71f8ad035bc593076fcef4effc4572280901d606/src/main/scala/mesosphere/marathon/tasks/TaskTracker.scala#L77.

This bug breaks the grace period because the calculation depends on the startedAt value. Hence, a task which is unhealthy from the beginning will never be killed because it is assumed that it's still in the grace period.
I will prepare a patch.


Add pattern support to PUBSUB CHANNELS (minor, fix)

Was omitted in #819

thanks!

@andymccurdy thank you, my bad :+1:


IR-631: Rename prisonId fields to location in reports and events

…to match api changes with respect to creating, retrieving and updating their models
Depends on api#206

@Mjwillis  this is the ui side, so none at all