Keras predict_step is not preserved across save and restore

System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow): No
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 20.04 and macOS
TensorFlow installed from (source or binary): binary
TensorFlow version (use command below): 2.6.0-dev20210329
Python version: 3.7 and 3.8

Describe the current behavior
When implementing custom prediction logic for Keras models using predict_step as explained here, saving and restoring the Keras model with the saved model format ignores the custom prediction logic. Unfortunately the code silently fails and doesn't inform the user that this is not supported, which could lead to detrimental bugs.
The issue is explained in detail with a minimal example in this colab notebook.
I know I can save a custom serving function using
class MyModel(tf.keras.Model):
    @tf.function(input_signature=[tf.TensorSpec(shape=[None], dtype=tf.string)])
    def serve(self, data):
        ...

as described here.
But I feel the current behaviour breaks with user expectations since the saved model format is now the default saving format but doesn't support all of the features and might silently fail resulting in unexpected behaviour.
This makes it necessary for users to break the abstraction and start using low level TF APIs instead, which I think doesn't fit well with the progressive disclosure of complexity that Keras tends to strive for.
Describe the expected behavior
Keras models should preserve custom predict_step logic when saving and restoring models.
Standalone code to reproduce the issue
import tensorflow as tf
import numpy as np

class FullyConnectedModel(tf.keras.Model):
    def __init__(self, **kwargs):
        super().__init__(**kwargs)
        self.dense = tf.keras.layers.Dense(10)

    def predict_step(self, data):
        logits = self(data, training=False)
        return tf.argmax(logits, axis=-1)

    def call(self, inputs):
        return self.dense(inputs)

x, y = np.random.uniform(size=(128, 20)).astype(np.float32), np.random.randint(0, 10, size=(128))

model = FullyConnectedModel()
model.compile(optimizer="sgd", loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True))
model.fit(x, y, epochs=2, batch_size=32)

model.save("/tmp/model", save_traces=True)
reloaded_model = tf.keras.models.load_model("/tmp/model")

y_pred = model.predict(x)
reloaded_y_pred = reloaded_model.predict(x)

np.testing.assert_allclose(reloaded_y_pred, y_pred)

See this notebook for more information.

@lgeiger,
The way you load the model should be some thing like,
loaded_1 = keras.models.load_model("my_model", custom_objects={"CustomModel": CustomModel})
Please refer the Documentation of Saved Model for Custom Objects and check if the issue persists?
Also, the error that you got, AssertionError: Not equal to tolerance rtol=1e-07, atol=0 looks fine to me because the Shape of Data (y_test) and the Shape of Prediction will not be the same.
print(y_test.shape, y_test.dtype)
print(y_pred.shape, y_pred.dtype)

Output:
(1, 10) float32
(1,) int64

Please let me know if I'm missing something. Thanks!

@rmothukuru Thanks for the fast response. Reloading with custom objects still does not work in this case.
I updated the notebook to make the issue more clear and test your suggestion.
Also in general I thought the idea behind the saved model format is that models can be reloaded without the code. Which also works for custom functions (see this example) so me as a user would expect that this is also true for custom train, predict and test functions.
What is the reason that they are explicitly ignored during serialization? To me the only reason for this could be that they include information about the steps_per_execution since this is set during model.compile (and not during model.fit), but I think this could be worked around or moved to model.fit to allow proper saving of custom models.
For now it looks like the only valid workaround is not to use the {predict, train, test}_step features at all and rather rely on completely custom functions that are serialized properly which seems inconsistent to me and might not be possible for cases other than predictions:
class ServingFullyConnectedModel(tf.keras.Model):
    def __init__(self, **kwargs):
        super().__init__(**kwargs)
        self.dense = tf.keras.layers.Dense(10)

    @tf.function(input_signature=[tf.TensorSpec(shape=[None, 20], dtype=tf.float32)])
    def serve(self, data):
        logits = self(data, training=False)
        return tf.argmax(logits, axis=-1)

    def call(self, inputs):
        return self.dense(inputs)

Also, the error that you got, AssertionError: Not equal to tolerance rtol=1e-07, atol=0 looks fine to me because the Shape of Data (y_test) and the Shape of Prediction will not be the same.

It is correct that the shape of y_test and y_pred are not the same, but the shapes and dtypes of y_pred before and after reloading should be the same which isn't the case. I updated the notebook to make this more clear.

Was able to replicate the issue in TF nightly-2.6.0.dev20210603,please find the gist here..Thanks !

I'm having the same problem here.

Hi @lgeiger ,
This behavior is due to the fact that the predict_step method is designed for optimizing inference performance within the training loop rather than for model serialization and deployment.
To achieve custom prediction logic preservation across save and restore operations, you can use tensorflow's @tf.function decorator and input_signature to define a serving function within your model. This approach allows you to explicitly specify how inputs should be processed during inference.
Here i am providing updated gist for your reference.
Thank you!


Ergo 0.21 : New APIs and new calculate call for formulas

Changes

Unnecessary async's removed on low-level calls in ergo-engine
New support for formulas, aligned with drafting in markdown-transform
Various build fixes

Coverage increased (+0.8%) to 96.247% when pulling 9647b3e020882fbeaf09892da290f5115285e8aa on parserv2 into d765f6d0401bd9b414416fa4c4a78e35d43de62d on master.


Update "Synced"/"Syncing" to "Connected"

addresses # 1 of #857

we can also change the header from "Sync Status" to something else if that would make sense. maybe "Hotspot Status", or "Connection Status"?

following the discussion with chain gang, we'll use a chain variable to decide when to display light hotspot-related UI.
chain var: poc_challenger_type would be validator
cc @allenan

(meant to add this screenshot on friday)

noticed a potential issue where the online status is pulling from the API status: online which uses stale online logic. might need an API update on testnet to follow what will be available on mainnet (which is: if there is activity in the last 36 hours, the hotspot is online).

documenting for future: as of this PR, the way the status works is:

if hotspot.status.online === 'offline', it shows "Offline"
otherwise:

if the poc_challenger_type chain var is set to 'validator' it shows "Connected"
if the poc_challenger_type chain var is not set to 'validator', it'll show "Syncing" / "Synced" depending on block height


Pulsar Startup Script fixes to provide custom conf files for bookie s…

Motivation
There are two bugs in the start up script:-
a. While running bin/bookkeeper due to hardcoding in (bkenv.sh) we can't give a custom conf file with different zookeeper ports.
b. While running bin/pulsar configuration-store the script overrides the custom conf file with default if PULSAR_GLOBAL_ZK_CONF is not provided.
Changes
Fixed both bin/pulsar and conf/bkenv.sh to accept custom config files.

rerun java8 tests


GSD Request

--- GSD JSON ---
{
  "vendor_name": "Linux",
  "product_name": "Kernel",
  "product_version": "versions from v5.10 to before v5.10.101",
  "vulnerability_type": "unspecified",
  "affected_component": "unspecified",
  "attack_vector": "unspecified",
  "impact": "unspecified",
  "credit": "",
  "references": [
    "https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id=e057dd3fc20ffb3d7f150af46542a51b59b90127",
    "https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id=7b53d2204ce79b27a878074a77d64f40ec21dbca"
  ],
  "extended_references": [
    {
      "type": "commit",
      "value": "e057dd3fc20ffb3d7f150af46542a51b59b90127",
      "note": "introduced"
    },
    {
      "type": "commit",
      "value": "7b53d2204ce79b27a878074a77d64f40ec21dbca",
      "note": "fixed"
    }
  ],
  "reporter": "joshbressers",
  "reporter_id": 1692786,
  "notes": "",
  "description": "can: isotp: fix potential CAN frame reception race in isotp_rcv()\n\nThis is an automated ID intended to aid in discovery of potential security vulnerabilities. The actual impact and attack plausibility have not yet been proven.\nThis ID is fixed in Linux Kernel version v5.10.101 by commit 7b53d2204ce79b27a878074a77d64f40ec21dbca, it was introduced in version v5.10 by commit e057dd3fc20ffb3d7f150af46542a51b59b90127. For more details please see the references link."
}
--- GSD JSON ---

This issue has been assigned GSD-2022-1000359


ExpressionBodySyntax: don't report if multiple returns in when expressions (Closes #7829)

This PR fixes #7829, not reporting a ExpressionBodySyntax if there are multiple return expressions in when statements.
This code is now compliant:
    fun stuff(): Pair<String, String>? {
        return Pair(
            first = "",
            second = when {
                true -> ""
                else -> return null
            }
        )
    }

Warnings

:warning:
This PR is approved with no milestone set. If merged, it won't appear in the detekt release notes.

  Generated by :no_entry_sign: dangerJS against<PHONE_NUMBER>05fb9b29b239a03d896cafb65c4dcf


Fix unable to add the owner of an organization to a team

Fix #238

Task linked: QF-1681 Unable to add the owner of an organization to a team


Fix some issues reported by lintian

Fix some issues reported by lintian

Use secure copyright file specification URI. (insecure-copyright-format-uri)
Bump debhelper from deprecated 9 to 12. (package-uses-deprecated-debhelper-compat-version)
Set debhelper-compat version in Build-Depends. (uses-debhelper-compat-file)
Change priority extra to priority optional. (priority-extra-is-replaced-by-priority-optional)
Drop unnecessary dh arguments: --parallel (debian-rules-uses-unnecessary-dh-argument)

This merge proposal was created automatically by the Janitor bot.
You can follow up to this merge proposal as you normally would.
Build and test logs for this branch can be found at
https://janitor.debian.net/lintian-fixes/pkg/yubikey-personalization-gui/4946aaf5-b8e4-4791-9f2d-85b66c5edf63.
Debdiff
These changes affect the binary packages:
File lists identical (after any substitutions)
Control files of package yubikey-personalization-gui: lines which differ (wdiff format)

Priority: [-extra-] {+optional+}

No differences were encountered between the control files of package **yubikey-personalization-gui-dbgsym**
You can also view the diffoscope diff (unfiltered).

This merge proposal will be closed, since all remaining changes have been
applied independently.

This merge proposal will be closed, since the branch for the role 'main'
has changed to None.


Project list does not scroll

I will prb get time to look at this tomorrow, Im guessing its just the task-bar overlapping the entire app at the bottom, so bottom of left/right part need to be adjusted with bottom: Xpx; (same height as the taskbar)

:+1:

I have a scrollbar on windows, and the bottom in the css is correct
Is this a mac issue?

during testing being able to add same project multi able times is great, but this can cause so much issues if someone messes up

strange, not sure how I got this then:

unless you still get it you can close this one

Will keep an eye on it :+1:

Ahh, got it.
if you refresh while some of the projects are overflowed:

then the scroll bar works OK after the refresh.
But when you refresh when the scroll bar isn't necessary:

then the scroll bar is never shown after the refresh

Someone seems to be using my identity - as I do not remember ever doing this:

Can we resolve this issue (whether there is some other adriatic - or am I losing my mind), please?

I added this item to the list, or else there is no use for the checklist

strange that it shows your name though

I am not the reasons for adding this, but rather why does it show me as adding it?

No idea, this looks like a bug in Github

Must be - I never saw this before so, I became paranoid that we have a party-crasher here :-)

so scrollbar works when you add project, but does not work after refresh of montery, if its not needed? (even if new projects are added?)

the behavior is kind of strange. When Monterey is opened when some projects are overflowed, then the scroll bar is visible and keeps working when I resize the screen (all good). but when I maximize the screen by double clicking on the window, then the scroll bar is gone, and won't return when I make the screen smaller so that the projects overflow
When monterey gets opened without projects overflowing (there are only a few projects in a very large window) then the scroll bar is never visible

Here is a vid: http://screencast.com/t/wQagKNpCpb

well I know the grid compensate if window resize event triggers and it redraws if it gets larger then it was(quickfix I added a while back) , but looks like I need to check the scrollbars that I only show if needed in same event, Ill try and get it fixed in grid code today or latest during the weekend.
Would have loved more control when parent div did resized, like a event.

as a workaround, can we always show the scrollbar?

it might work as a quickfix until I fix the bug in the grid


etcdctl cluster-health reports unhealthy node as healthy

etcd version: v2.0.3
How to reproduce:

Start a 3 node etcd cluster.
Kill a node (hard poweroff the machine -- no clean shutdown)
run etcdctl cluster-health

Expected results:
etcd reports the kill node as unhealthy
Actual Results:
$ etcdctl cluster-health
cluster is healthy
member 5ae3067007f7fb85 is healthy
member 7931e79c0d8b47c5 is healthy
member 987146e8925f10e5 is healthy <- This node is powered off

Seems after 5 mins or so, the cluster-health command reports the node unhealthy:
$ etcdctl cluster-health
cluster is healthy
member 5ae3067007f7fb85 is healthy
member 7931e79c0d8b47c5 is healthy
member 987146e8925f10e5 is unhealthy

@kelseyhightower
The current detection is not very reliable since it is using the old 0.4.x endpoint. The member healthy checking is based on the underlay tcp connection. If you just power off the machine, the connection will not be dropped immediately.
It command achieve our current goal for detection the healthiness after migration. We can improve it later.

What will the approach be on this? Are we going to add a threshold for RPC latency and then mark something as unhealthy at that point?

I am moving this to 2.2.
This is more in line with our goal for 2.2: etcdctl uses etcd/client and new endpoints if possible.

This is kind of duplicated with #2711 and @yichengq is working on it. So I am closing this one if favor of #2711.


Fix tests for Basic JavaScript: Write Reusable JavaScript with Functions

fix original log() method restoration
fix case that other userscripts(or extensions) can affect console's output and therefore can fail the tests

[x] I have read freeCodeCamp's contribution guidelines.
[x] My pull request has a descriptive title (not a vague title like Update index.md)
[x] My pull request targets the master branch of freeCodeCamp.
[x] None of my changes are plagiarized from another source without proper attribution.
[x] My article does not contain shortened URLs or affiliate links.

@T1mL3arn Just so we are clear on your intent, what is an example of a case where "other userscripts(or extensions) can affect console output and therefore can ruin the test"?

@RandellDawson This is what I think it was:
Any userscript or browser's extension can log info to console. Sometimes this happens just before tests, therefore even if the user's print is correct the tests gets an userscript's output and fails.
That happened to me - I have some scripts which log messages to console and I was not able to pass this challenge. But after disabling all problem scripts problem was gone.
So now
if(message === 'Hi World')  hiWorldWasLogged = true;
assert(hiWorldWasLogged, 'reusableFunction should output "Hi World" to the dev console');

ensures that user's function prints what is expected and tests just check this flag.
I can try to write minimal example, if that explanation is not enough.

LGTM, but needs another local tester.

LGTM too except that `logOutput' should be removed since it is no longer used in the tests.  It does turn up in the teardown, but I think the teardown should just be
uncapture();

as the two IIFEs do nothing, as far as I can see.

@T1mL3arn Thank you for your contribution to the page! 👍
We're happy to accept these changes, and look forward to future contributions. 📝


Rename 2D NoiseTexture to NoiseTexture2D

This rename would be helpful to have done before implementing https://github.com/godotengine/godot-proposals/issues/4946
However, I think this stands on its own. Since it inherits Texture2D it makes sense to have the name end in Texture2D, so I'm opening a PR just for this. I briefly discussed this on RocketChat.

What about all the other *Texture classes though, like AnimatedTexture or CanvasTexture?

@akien-mga I don't know about each case, but in general that makes sense and I would be in favor of renaming some of those too. For this PR, in addition to the Texture2D inheritance, I am focused on this case because 3D noise is known to be a desired feature so making the 2D one explicit makes more immediate sense.

Discussed it with @reduz who seems to be fine with the change:
<Akien> @reduz Thoughts on #64864 ? My main doubt is that it adds more inconsistency in the naming of our texture classes.
Currently, most classes inheriting Texture2D don't use the 2D suffix:
[img of the classes inheriting Texture2D]
<reduz> @Akien it would make sense if we think a NoiseTexture3D makes sense
which may do
the others are clearly 2D only
<Akien> Yeah this PR is in support of a proposal to add NoiseTexture3D
My doubt is whether we'll ever want to also add AnimatedTexture3D or something like that.
<reduz> probably not, and that reminds me that i need to remove ProxyTexture
its not so useful now and its not so efficient in Vulkan


Support budget for categories

It's useful to keep Lighthouse categories in check, especially those, that rely on static page analysis, like accessibility, seo, and best-practices. Decrease of the score is often a sign of a quality regression.
This feature will allow analyzing pages with Lighthouse in CI and will help to make projects like https://github.com/treosh/lighthouse-ci-action more useful.
Proposed format:
[
  {
    "path": "/*",
    "categories": [
      {
        "id": "accessibility",
        "minimumScore": 1
      },
      {
        "id": "seo",
        "minimumScore": 0.9
      },
      {
        "id": "best-practices",
        "minimumScore": 0.8
      }
    ]
  }
]

The format is open for debate. The proposed one is inspired the current Lighthouse report on categories with id and minimumScore value between 0 and 1.

Huge +1 to this. This is a great idea. Thanks for opening this issue.
Happy to bike shed over format :)
This is the format that I'm currently leaning towards. Very interested in feedback on this:
[
  {
    "path": "/*",
    "lighthouse": [
      {
        "category": "accessibility",
        "budget": 1
      },
      {
        "category": "seo",
        "budget": 0.9
      },
      {
        "category": "best-practices",
        "budget": 0.8
      }
    ]
  }
]

minimumScore --> budget
budget would match the syntax used elsewhere in budget.json to declare budgets. However, the one possibly weird thing about this is that in this context a budget would represent a minimum threshold, but everywhere else it is a maximum threshold. I have very mixed feelings on this and I can also see score or mimimumScore making more sense here. @patrickhulce might also have opinions here since we were just discussing something similar here.

categories --> lighthouse
If Lighthouse were to change how it works (e.g. no categories, or a significant feature that isn't categories), this would be more flexible.

Integer vs. decimal (for budget/score)
I had mixed feelings at first but I think I'm also leaning towards using decimal format here. I think which format seems more intuitive to a user probably depends on how they use Lighthouse: if they use the HTML report an integer probably makes more sense; if they consume the JSON version then decimal probably makes more sense. However the fact that LighthouseCI also uses the decimal score format probably pushed me over the edge for preferring decimal over integer.

Specific Responses/Thoughts

However, the one possibly weird thing about this is that in this context a budget would represent a minimum threshold, but everywhere else it is a maximum threshold.

The name here to me is highly dependent on how the statisticalApproach shakes out. If it stays as the current proposal (applies with file-scope and is named optimistic/pessimistic) then I think we're fine to continue using budget and put the onus on the budget validator logic to correct for this flip of min/max desirability.

categories --> lighthouse

I like this change a lot 👍 not only for the flexibility of Lighthouse assertions into the future but also the siloing of lighthouse-specific config going into an explicitly named property, just for it (see meta point below).
Meta Point
I'm not sure yet if I love lighthouse-specific assertions in the budgets.json format :)
On the one hand, I see budgets.json as an important generic web tooling format that could encourage every tool to start caring about webperf and be able to directly leverage the existing budgets.json files out there is a low barrier to entry. I'm a little concerned that including lighthouse-specific things in there might muddy that message a little. When we have alternative, lighthouse-specific solutions in lighthouse-ci on the horizon, feels a tad like an unnecessary risk.
On the other hand, I'm a Lighthouse engineer that would love to see these aspects of web quality taken into consideration as widely as possible, so inclusion in a generic budgets format is awesome! :)

@khempenius  I agree with all your suggestions. Moving LH logic to a separate section makes sense. budget is a better name, and min/max logic should go from the context. Scores and sizes have reverse nature, but conceptually the idea of budget is always to cap the worst case.
Meta-level
@patrickhulce initially, I thought about budget.json as a tool to run assertions for Lighthouse and programmatically generate snapshots to capture the state of the URL. But now, I see lighthouse-ci assertions, and also not sure if budget.json should have LH specific logic, or to be purely generic format that follows web standards.

👍 Would love to see this happen!


Added argocd pkg

added argocd package

@kononovn I just noticed this PR is adding both ApplicationBuilder and Builder structs into. I think if multiple structs are in the same pkg, we should make both specific (i.e. ApplicationBuilder and ArgCDBuilder). WDYT?

@kononovn I just noticed this PR is adding both ApplicationBuilder and Builder structs into the argocd pkg. I think if multiple structs are in the same pkg, we should make both specific (i.e. ApplicationBuilder and ArgCDBuilder). WDYT?

@trewest No need considering package name. Code usage will look like:
arogcd.NewBuilder
argocd.ApplicationBuilder
We do have argocd in package name


cloudsql/mysql: add support for Cloud Run

Updates the Cloud SQL sample for simultaneous compatibility with App Engine and Cloud Run.

This PR shifts the app to be a go module.
I think the failure is:
+ go vet ./cloudsql/...
go: warning: "./cloudsql/..." matched no packages
no packages to vet

Could there be a problem with go vet discovering a nested module?

@tbpg that doesn't change that tests are failing, do you have any insights on the go vet hypothesis?

Yes, it seems go vet doesn't check modules outside the main module...
➜  go vet ./cloudsql/...
go: warning: "./cloudsql/..." matched no packages
no packages to vet
➜  go vet ./cloudsql/mysql/database-sql/...
go: directory cloudsql/mysql/database-sql is outside main module
➜  cd ./cloudsql/mysql/database-sql
➜  go vet ./...
➜

I'd vote merge this and file an issue to make sure it gets fixed.


Explore DEM processing solutions for thalweg burn "notch" artifact in SRC

Longitudinal thalweg DEM burn-in process is creating a "notch" in the channel cross section. This leads to artifacts in the REM grid and SRC output.

This screenshot is for the MS River near Eudora and Greenville.

To Do

[ ] Run thalweg_drop_check.py

[x] Create feature branch
[ ] Debug for threshold argument not making it into compare_thalweg function

[ ] Clean up /outputs/single_pixel_inundation/ to save on VM storage
[ ] Identify HUCs from single_pixel_inundation run where there are large drops where we can do some testing (qualitative assessment)
[ ] Ideas for correction
refine/turn off lateral thalweg adjustment
[ ] Get Carson access to production server
[ ] Run tests/eval on production server

@CarsonPruitt-NOAA @BrianAvant-NOAA Carson should have access to the Cahaba API right? Will he need access to the production machine, too?

@BradfordBates-NOAA @CarsonPruitt-NOAA He should have access to the API but it isn't configured to run the sequence of events we need for this task.

#679 fixed the REM but did it fix the SRC?


FINERACT-1498: Switching from Flyway to Liqubase migrations

Functionality description
Full change from Flyway migrations to Liquibase migration. The Liquibase migration files have been generated using Liquibase CLI and based on the latest develop commit (c66415dbfdcb54ff9dada4fdcee603fba7a151e2). There have been a couple manual changes to the changesets in the generated migrations since there were a few issues where the ordering of changes wasn't right or when the code was depending on some certain database naming for unique key constraints.
The change includes an automatic forward schema upgrade from Flyway to Liquibase. There's no need to do anything manually. Upon startup, Fineract will decide on the status of the database - whether it's empty, whether it was running Flyway on the latest 1.6 Fineract version or if the database state represents a previous version of Fineract.
The automatic upgrade works in 3 different ways depending on the scenario

If the database is completely empty, it's going to apply all changesets from the Liquibase changelogs
If the database is contains the changes from the last Flyway migration scripts included in Fineract 1.6. In that case, a seamless upgrade is going to happen to Liquibase
If the database contains Flyway changes but it's not running the latest migration scripts from 1.6 - i.e. the Fineract version being ran is pre-1.6. In that case the application startup will fail, notifying the operator in the logs that first they shall upgrade to 1.6 and then to 1.7

Testing
On top of the tests written for the changes the following cases have been checked manually, all of them passed successfully

Starting from an empty db
Starting from an empty db and then applying one new changeset
Starting from an existing flyway migrated db
Starting from an existing flyway migrated db and then applying one new changeset
Starting from an existing flyway migrated db which is not running the latest structure on tenant store, app should fail
Starting from an existing flyway migrated db which is not running the latest structure on tenant, app should fail
Starting from an existing flyway migrated db which is not running the latest structure on tenant but upgrades to 1.6 then applies Liquibase changes

What's still missing

README.md update
Squashing the commits

Checklist
Please make sure these boxes are checked before submitting your pull request - thanks!

[x] Write the commit message as per https://github.com/apache/fineract/#pull-requests

[x] Acknowledge that we will not review PRs that are not passing the build ("green") - it is your responsibility to get a proposed PR to pass the build, not primarily the project's maintainers.

[x] Create/update unit or integration tests for verifying the changes made.

[x] Follow coding conventions at https://cwiki.apache.org/confluence/display/FINERACT/Coding+Conventions.

[x] Add required Swagger annotation and update API documentation at fineract-provider/src/main/resources/static/api-docs/apiLive.htm with details of any API changes

[x] Submission is not a "code dump".  (Large changes can be made "in repository" via a branch.  Ask on the developer mailing list for guidance, if required.)

FYI our guidelines for code reviews are at https://cwiki.apache.org/confluence/display/FINERACT/Code+Review+Guide.

Just as a FYI for everyone who comes across this PR: the intention is to merge this AFTER we've done the 1.6 release

@vidakovic
This seems very interesting to me. I would love to see this in action. Interesting indeed

Just FYI, I'll squash the commits as soon as I get some approvals. Let me know if you have any comments/questions.


Error when Trying to Share Screen With Chrome

I cloned the repo, changed the externally connectable URL, created an extension and pasted the extension  ID on my config.js file.
But when some of my users try to use the extension the get a message saying "Permission Denied, failed to install extension".
This is happening on Windows 10 with Google Chrome 59.0.3071.115.

Did you publish the extension in chrome web store?


fix: //docs tests in MacOS

Fixes #138

@thesayyn similar to #136, maybe this PR could go in and postpone some further investigation as to what's breaking in MacOS for the //docs tests
I tried upgrading some dependencies and couldn't get it to work. Maybe it's just my Mac? :-?


(PUP-4347) Add test for the File resource ignore attribute

This commit adds 5 tests for the ignore attribute of the File resource.

CLA signed by all contributors.

LGTM.  waiting on travis


Add a test for correct updating of sorted Results after insertion

This passes on master but was broken in 0.93.2 and didn't have any tests covering it. Closes #2166.

:+1:


Enable vexriscv-smp on a200t / nexys-video board

Any reason this isn't enabled already?

Originally, when adding this test, there was an issue that now should be solved in the symbiflow-arch-defs (related to the CARRY4 CHAIN packing). I will try and see whether we can enable this test for all the other toolchains.


Modification to the way we define the cost

Thanks to the advice of @mgebser  , we need to modify the way we handle the weights of the different costs.
Also we need to handle when there is no solution in a smarter way (not just an upper limit of the iteration)

We now need to determine throught benchmark if it is better to have different cost level or if it is best to have a weighted sum


speed up the snake_case/camelCase conversion

part of: https://github.com/kiali/kiali/issues/6032
speed up the snake_case/camelCase conversion

ran a couple of the molecule tests that would flush any problems out - all pass successfully. This looks good.

all molecule tests pass on minikube. this is good to go


Return error level != 0 for parsing errors.

When the command line script fails to parse the PCB we must indicate it
using an error level != 0.
A simple test PCB to reproduce it is:
(kicad_pcb (version 20171130) (host pcbnew 5.1.5+dfsg1-2~bpo10+1))

Also note that returning 1 is confusing because this is the value returned for unhandled exceptions.
The value of 2 is used by argparse.
IMHO the script should return 3, 4 and 5 for the 3 different errors that is returning 1.
I can do a PR for this if you want.

Thanks for the PR, I agree with general idea but implementation can be improved.

Returning boolean is not good practice, either introduce a status struct with code and message, or just throw a ParsingException (can be added if there is no suitable python built in). I prefer exception route.
Print the message from 1 before exiting script.
Apply same logic to run with dialog as well.

I used an exception, and I also used logger.error to print the exception message.


GitHub Actions Enhancements

What would you like to share?
To add Kubescape YAML actions file to the project

Thank you for raising but there is no kubernetes directory is this project for deployment, so it does not make sense to add the GitHub Action here to check it - please only add it to projects that have the kubernetes config

I will keep that in mind Eddie.


Feedback on Step5 Exercises

All the other steps from step1 through step4 increase in difficulty as you progress. I found step5 to be almost too easy. There's nothing much to be done.
I think it would benefit from having a couple of more problems to solve.

Yes Step5 is an anticlimax. often with FS2 the challenge is figuring out the right combinator, rather than writing a lot of code. I didn't think the participants were going to come up with it on their own, so with the hint it ends up rather easy.  At this stage Im just going to let it be, since there's more content than they'll cover in 4 hours I think..


Generics lost when accessing a property

TypeScript Version:  4.1.0-dev.20201015
Search Terms:
lost generic types, generics property access, generic function invocation
Code:
declare const abc: {
  a: {
    val: number,
    fn: (a: number) => string
  },
  b: {
    val: string,
    fn: (a: boolean) => number
  }
};

type Param<T extends 'a' | 'b'> = Parameters<typeof abc[T]['fn']>[0];
type Model<T extends 'a' | 'b'> = ReturnType<typeof abc[T]['fn']>;

const tester = <T extends 'a' | 'b'>(type: T, p: Param<T>) => {
  const a = abc[type]; // Generic is OK here
  const b = a.val; // Generic lost and becomes union :(
  const bb = a.val as typeof abc[T]['val']; // Explicit cast - Either number or string but in generic context
  // b expected to be bb

  const c = a.fn // generic lost and becomes union :(
  const d = c(p); // accepts never and returns string | number - generic lost :(
  const cc = a.fn as (a: Param<T>) => Model<T>; // Explicit cast
  const dd = cc(p); // Either number or string but in generic context
  // c expected to be cc
}

Expected behavior:
The example is based on two problems which are essentially the same.
The first is more awkward - abc[type] retains the generic type and doesn't expand it,
while for some reason abc[type].val loses the generics. I would expect it to be the same type as doing typeof abc[T]['val']
especially when abc[type] works correctly.
The second example is concerning function invocation too but the problem's base is the same. When accessing abc[type].fn the generic type is lost and becomes a functions union which isn't invokable (parameter becomes never).
The wanted result is actually (a: Param<T>) => Model<T> retaining the generics and allowing invocation.
Actual behavior:
Concerning the function i kind of understand the design limitation but for abc[type].val it seems like a bug.
Playground Link: https://www.typescriptlang.org/play?ts=4.1.0-dev.20201015#code/CYUwxgNghgTiAEYD2A7AzgF3lARmAXPAN4CwAUPNoaRZfAG5QSEoCuAtjiDADTl3wAZikIAKKCw5cYASngBeAHzxMMAJYoA5v3gBfPrRzUdlRsxUZ1WgwOFiJ8HEiQQQUFHKXw2nbjt3kugDc5OQYAJ4ADggACrBQ7AA8ACrwIAAeGCAowGjwAORQ+fAAPgU4+cry8HEwCSBZMGiJEdFIgth4ANrJALpd+cL5vYpdAAy9IWStCACySKAQKWmZ2bkFRaXllQrwAEoNrDAoyVEgLWftnWA9-YMow4pT5MjoWFmY3LvLGVk5eYVimV8hVFKIZoRkjx4JFCLUEilFJ5lDRKK9MNhdrgbjNJvAAPT4+AAcWy3DUYHgajyAHkANLwAAW3BAOnRWBwWIAdGYggSiaSUOTKRAkBj3MBHOAkOwQHlWCg1Kh4PhRGzUBicJzqlAeUxsHkZldsbcBmZhnzCfAAKLpSIQClqLBgKAYgC0NqdzJg3ikXyQPtUGk0jlYWA08E0ZPUlNeWUyOitnIy0TAWUlGCQUscOFCtHZiG5wn5kejFPgovFOSlyFl8sVytV6re8El1TAokiMktRKgYDAIEiGDyQvoXwl8DgGCO6AsVhDZR80ngHqjQpjFbFWCb+Y1zspOq5xdd8HEcPiSWSSIUynmi0RPZtdodYCdiFdGGbGOAbcQHa7j7Wl6XxLv6gaWMGobhigpbruWca-ImRKUim4DpvAmbZv2gTkEAA
Related Issues: https://github.com/microsoft/TypeScript/issues/36737 (although seems the first part of mine isn't related)

This is the intended behavior - lookup types are generally only preserved when it's not possible to resolve them through a constraint on that type parameter. The alternative universe where everything is processed in terms of deferred lookup types is really unpleasant to work in since there's no reasoning you can apply to a deferred operation. In general this only decreases completeness rather than inducing unsoundness, so overall it's a good compromise done on purpose.

@RyanCavanaugh But isn't it the idea of generics? I could pass a union directly and then handle the type problems with type casts. But i decided to go for generics saying that everything is based on it. I agree that those types can become ugly but maybe that's another point of how to display such types. The union fallback seems wrong... The only time i would expect it is if everything was the same type which means the generic parameter had no meaning anymore. But there is a big difference between string | number and the generic based type which i think worth it compared to type casts.
The behaviour here should be for regular union and not for generics imo. Or at least i should have a way to specify that this is what i mean to the compiler. Maybe both can be supported somehow based on the scenario?


openapi 3.1 - support tuples

Open API 3.1 supports defining tuples via Json Schema prefixItems.
Given the following TypeSpec:
model Foo {
  tuple: [string, string];
}

We can emit an Open API 3.1 schema that looks like this:
Foo:
  type: object
  properties:
    tuple:
      type: array
      prefixItems:
        - type: string
        - type: string
  required:
    - tuple

This matches what our json schema emitter currently emits.
For reference, our Open API 3.0 emitter does not support emitting tuples due to the lack of prefixItems and instead outputs this:
Foo:
  type: object
  required:
    - tuple
  properties:
    tuple:
      type: array
      items: {}

Pri: 2
est: 5


ske: Atlas module that automatically packs a list of arbitrary images.

Original issue:
~mrms/tophat#42
Initial message:

@skejeton are you okay with atlas.Atlas.pack? If so, please close this.

Is the cell size sparse or constant?

It's constant, since tophat atlases don't allow variable cell sizes anyways.

I think this works for games with blocks, although I was more thinking for atlases of texture regions, where the ordering doesn't matter. I think this is good enough for now though, if anything it could be an extension, and someone can make their own packer.


Add external,  Add parameter -Name to examples (required)

Type of Change

[ ] Bug fix (non-breaking change, fixes #)
[ ] New feature (non-breaking change, adds functionality)
[ ] Breaking change (effects multiple commands or functionality)
[ ] Ran manual Pester test and has passed (`.\tests\manual.pester.ps1)
[ ] Adding code coverage to existing functionality
[ ] Pester test is included
[ ] Nunit test is included
[x] Documentation
[ ] Build system

Purpose
Improve Documentation / examples

awesome, thank you!


Add pushJobSpec to ingestion yml files

Added pushJobSpec settings for ingestion yml files so they will work. Updated staples schema to create a 'Staples' table instead of 'staples' so the TDVT tests stop using a table alias (not supported by pinot).

Change the title of the PR to "Add pushJobSpec to ingestion yml files and Update staples schema"


Database migration tools for graphback

Summary
We can separate concerns of handling database by using preexisting database tools which will handle database migration.
TODO
List out database tools we can employ to our usecase

After investigation I think we need prototype for Postgres and MySQL based migrations using knex

Requirements:

Support SQL migration (in form of README.md file)
Support direct database migration for dev
React-based on environment abstraction
React-based on generated database mapping file.

We going to build our own migration tooling basing on GraphQL https://github.com/aerogear/graphql-relx/issues/3

Reopening as we trying to apply some minimal changes inside graphback first before adding things to relix

@craicoverflow Closing as we decided to use our own implementation. There is no reliable migration tool for the moment in js community that will work nicely with graphql.


Error running full sets with vcs backend.

I was running all sets with vcs on aims4 returning following error.
This error was reproduced by @golaz on Nersc and @zshaheen on Aims4.
Traceback (most recent call last):
  File "/export_2/zhang40/anaconda2/envs/acme_diags_env_v1/bin/acme_diags_driver.py", line 4, in <module>
    __import__('pkg_resources').run_script('acme-diags==1.0.0', 'acme_diags_driver.py')
  File "/export_2/zhang40/anaconda2/envs/acme_diags_env_v1/lib/python2.7/site-packages/pkg_resources/__init__.py", line 742, in run_script
    self.require(requires)[0].run_script(script_name, ns)
  File "/export_2/zhang40/anaconda2/envs/acme_diags_env_v1/lib/python2.7/site-packages/pkg_resources/__init__.py", line 1503, in run_script
    exec(code, namespace, namespace)
  File "/export_2/zhang40/anaconda2/envs/acme_diags_env_v1/lib/python2.7/site-packages/acme_diags-1.0.0-py2.7.egg-info/scripts/acme_diags_driver.py", line 120, in <module>
    parameters = cdp.cdp_run.multiprocess(run_diag, parameters)
  File "/export_2/zhang40/anaconda2/envs/acme_diags_env_v1/lib/python2.7/site-packages/cdp/cdp_run.py", line 23, in multiprocess
    results = bag.map(func).compute(num_workers=parameters[0].num_workers)
  File "/export_2/zhang40/anaconda2/envs/acme_diags_env_v1/lib/python2.7/site-packages/dask/base.py", line 99, in compute
    (result,) = compute(self, traverse=False, **kwargs)
  File "/export_2/zhang40/anaconda2/envs/acme_diags_env_v1/lib/python2.7/site-packages/dask/base.py", line 206, in compute
    results = get(dsk, keys, **kwargs)
  File "/export_2/zhang40/anaconda2/envs/acme_diags_env_v1/lib/python2.7/site-packages/dask/multiprocessing.py", line 177, in get
    raise_exception=reraise, **kwargs)
  File "/export_2/zhang40/anaconda2/envs/acme_diags_env_v1/lib/python2.7/site-packages/dask/local.py", line 521, in get_async
    raise_exception(exc, tb)
  File "/export_2/zhang40/anaconda2/envs/acme_diags_env_v1/lib/python2.7/site-packages/dask/local.py", line 290, in execute_task
    result = _execute_task(task, data)
  File "/export_2/zhang40/anaconda2/envs/acme_diags_env_v1/lib/python2.7/site-packages/dask/local.py", line 271, in _execute_task
    return func(*args2)
  File "/export_2/zhang40/anaconda2/envs/acme_diags_env_v1/lib/python2.7/site-packages/dask/bag/core.py", line 1405, in reify
    seq = list(seq)
  File "/export_2/zhang40/anaconda2/envs/acme_diags_env_v1/lib/python2.7/site-packages/dask/bag/core.py", line 1565, in map_chunk
    yield f(*a)
  File "/export_2/zhang40/anaconda2/envs/acme_diags_env_v1/lib/python2.7/site-packages/acme_diags-1.0.0-py2.7.egg-info/scripts/acme_diags_driver.py", line 61, in run_diag
    single_result = module.run_diag(parameters)
  File "/export_2/zhang40/anaconda2/envs/acme_diags_env_v1/lib/python2.7/site-packages/acme_diags/driver/zonal_mean_2d_driver.py", line 227, in run_diag
    diff, metrics_dict, parameter)
  File "/export_2/zhang40/anaconda2/envs/acme_diags_env_v1/lib/python2.7/site-packages/acme_diags/plot/__init__.py", line 41, in plot
    plot_fcn(ref, test, diff, metrics_dict, parameter)
  File "/export_2/zhang40/anaconda2/envs/acme_diags_env_v1/lib/python2.7/site-packages/acme_diags/plot/vcs/zonal_mean_2d_plot.py", line 106, in plot
    vcs_canvas.plot(test, template_test, test_isofill)
  File "/export_2/zhang40/anaconda2/envs/acme_diags_env_v1/lib/python2.7/site-packages/vcs/Canvas.py", line 2690, in plot
    a = self.__plot(arglist, keyargs)
  File "/export_2/zhang40/anaconda2/envs/acme_diags_env_v1/lib/python2.7/site-packages/vcs/Canvas.py", line 4020, in __plot
    returned_kargs = self.backend.plot(*arglist, **keyargs)
  File "/export_2/zhang40/anaconda2/envs/acme_diags_env_v1/lib/python2.7/site-packages/vcs/VTKPlots.py", line 680, in plot
    vtk_backend_grid, vtk_backend_geo, **kargs))
  File "/export_2/zhang40/anaconda2/envs/acme_diags_env_v1/lib/python2.7/site-packages/vcs/vcsvtk/pipeline2d.py", line 296, in plot
    self._updateVTKDataSet(plotBasedDualGrid)
  File "/export_2/zhang40/anaconda2/envs/acme_diags_env_v1/lib/python2.7/site-packages/vcs/vcsvtk/pipeline2d.py", line 347, in _updateVTKDataSet
    dualGrid=dualGrid)
  File "/export_2/zhang40/anaconda2/envs/acme_diags_env_v1/lib/python2.7/site-packages/vcs/vcs2vtk.py", line 547, in genGrid
    wc, [xm, xM, ym, yM], wrap))
  File "/export_2/zhang40/anaconda2/envs/acme_diags_env_v1/lib/python2.7/site-packages/vcs/vcs2vtk.py", line 2017, in getWrappedBounds
    assert (x1 < x2 and y1 < y2)
AssertionError

from @zshaheen
Running with multiprocessing and sets=[4] works (sets=[3,4,5,7,13] causes the error)

@doutriaux1 : we were trying to do a final check before the v1 release of ACME Diags and ran into this problem. Any idea what this might be?

@golaz I'm looking into it, I need to understand exactly what dask does, from the look of it it's almost as if concurrent process share/alter each other memory. It is very worrisome. @danlipsa we're passing the numpy arrays pointers to VTK, would it be possible that somehow 2 VTK instance runnnig at same time share the same memory space? The other thing it could be would be mesa and llvm, @zshaheen was going to see if he can reproduce the issue w/o using mesalib. It's possible that LLVM's magic interfers with multiprocessing.

@doutriaux1 Not sure I understand this. Are we plotting from multiple processes or threads  running on the same machine? Which one is it processes or threads?

@danlipsa look at the issue in vcs referenced above. It is not threads, it is multiprocessing via dask. I got a small example there

@doutriaux1 Multiple processes/nodes compute the data but in the end there is only one process calling vcs isn't it?

@danlipsa no many processes each producing a plot/png see example

Charles made an issue on the vcs repo here.
Well, here's an example of how it's done:
Say we have 32 processes and 256 plots to run. This means that each process will call the plot function here 8 times. So each plot that's generated is not completely independent. There is some shared memory, which might be what's causing the error.

@doutriaux1 @zshaheen I would try to run this serially with npartitions=1. Maybe its not the paralellism but simply vcs has problems with one of the data it tries to plot.

Lee had the same problem: full set run could not complete with 'vcs'.
We will plan a workaround hope to temporary fix this issue:
from @zshaheen
"It’s only set 4 (zonal mean 2d) that breaks w/ multiprocessing.
If the user does multiprocessing = True, we can just have this set run using mpl, like how we do for the cosp histogram.
If the user does run it in serial, the set 4 will be made in vcs, not the forced mpl."

This issue is fixed when we have 1 plot per processes. This is done by default. Testing instructions were passed on to the vcs and vtk people.


numberInput have error when the value is 0

Description
if the value is 0 , you can not click the add button, i think 0 is false and cause this bug.
Steps to reproduce

Go to '...'
Click on '...'
Scroll down to '...'
See error

ILLA Builder Version
1.2.3
Browser
No response
Operating System

[X] macOS
[ ] Windows
[ ] Linux

Additional Information
No response

Hey There:
Thank you very for much the issue. We already merged your code. Much appreciated!
You can join our Discord Community for further communication!  Here is the Discord: https://discord.com/invite/illafamily.
Best,
ILLA Team


[bug report] can't extract some ramdisk on windows

Basic information

Am I using debug build: NO

OS and machine: Windows 10 22H2 19045.4529

ZIP file name of the magiskboot binary I am using:  magiskboot-e159716-release-windows-mingw-w64-msvcrt-i686-standalone.zip

Description of the bug

when i tried to unpack ramdisk.cpio，it return a error
Expected behavior

extract all files in ramdisk.cpio
Bug hehavior

magiskboot.exe cpio ./ramdisk.cpio extract
Loading cpio: [ramdisk.cpio]
Extracting entry [acct] to [acct]
Extracting entry [apex] to [apex]
Extracting entry [bin] to [bin]
unknown entry type
Failed to process cpio

Extra links

ramdisk.cpio

offtopic: is that file uploaded by u (?)

ramdisk.cpio.zip

thanks for reporting , will be looking into it when i have time

currently i suspect its a bug in my winsup stub code

can't produce on linux, issue is windows only

can't produce on linux, issue is windows only

yes,i have tried macos and linux(both x86_64 and arm64).both of them can extract ramdisk.cpio.only windows can't handle this file

interesting

btw maybe try to use the cygwin build cygwin-test (release) in the meanwhile

well, cygwin-build works fine.it extract all files in cpio.
but this build has too many dll and has no win32 version.i will try to pack all of them into one exe
thanks for your reply!

i can make a standalone cygwin build, that only depend on cygwin1.dll

if you want

i can make a standalone cygwin build, that only depend on cygwin1.dll

that's really a great news!
i think i need it
thank you very much!~

a new problem:
command:
magiskboot cpio ramdisk.cpio "extract /system/bin/init init"
output:
Loading cpio: [ramdisk.cpio]
Extracting entry [system/bin/init] to [init]
thread '<unnamed>' panicked at 'strerror_r failure', library/std/src/sys/unix/os.rs:127:13
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

using binary form this

sad


Feature request - datagen.flow_from_files

So as you know there is already datagen.flow_from_directory which takes the images from whatever order that is in the directory. The function in itself is extremely useful in that we can rotate, zoom, resize the image.
However, it will be extremely useful if:

We have a datagen.flow_from_files function where we specify the files as say an array. This will help us resample images that have a under-represented class. It will also help with randomising the images so that we aren't limited to how they are ordered within the 'positive'/ 'negative' class folders (eg. in the cats vs dogs example in the blog)
(optional) A second function that lets us control the amount of zoom, rotation etc. Right now I do not know how much it zooms in, or rotates at each epoch when I specify the values as (0.1, 10). This would be useful if I could create my own generator function so that I can control the levels of zoom, rotations etc. with a for loop in my generator function.

I am happy to contribute if someone could point me to the code where I can create such a function.

you can find the flow_from_files function here
https://github.com/fchollet/keras/blob/ec39c35975906801d49ceab931f6b35deab8f438/keras/preprocessing/image.py#L463
it used the DirectoryIterator, which you can find here https://github.com/fchollet/keras/blob/ec39c35975906801d49ceab931f6b35deab8f438/keras/preprocessing/image.py#L898

I made a proposal that would support this at https://github.com/fchollet/keras/issues/6538#issuecomment-302835861, let me know what you think over on #6538.


ModuleNotFoundError: No module named 'readline'

when I finish the installation and try to run the framework I'm getting this message of error:
┌──(root💀kali)-[/home/kali/Downloads/ghost-main]
└─# ghost
Traceback (most recent call last):
File "/root/.ghost/./ghost", line 33, in
import readline
ModuleNotFoundError: No module named 'readline'

#50

#50


dm_thinpooldev on RHEL

This is more of a documentation problem because it took me too long to work through this.
The option dm_thinpooldev (added in #342) on RHEL has an unmet dependency kindof.  If you use this setting, docker-storage-setup assumes it really must be there and honor it even if it hasn't been created yet.  If you don't supply this option and it's using devicemapper (default on RHEL), it uses loopback drivers mapped through devicemapper.  In the documentation for setting up docker on RHEL, it very states rather bluntly you shouldn't use that in production.  This is because of scaling and performance issues related to the loop devices.
This means to break the cycle you have to have created the proper thinpool for docker ahead of time.  The puppetlabs/lvm module can do it if you use the logical_volume resource, not class, and set the type to thin-pool.  Alternatively you could do a one-time exec to do the create for you that would look something like this:
lvcreate -y -L 10G -n docker-poolmeta vg0 && lvcreate -y -L 101G -n docker-pool vg0 && lvconvert -y --zero n -c 512K --thinpool vg0/docker-pool --poolmetadata vg0/docker-poolmeta

Not completely true. There seems to be a dependency missing... This module should create the /etc/sysconfig/docker-storage-setup file, then run docker-storage-setup. docker-storage-setup will then create the thinpool for you just nicely. Since somewhere in the last month the docker-storage (not the -setup one) is being managed before enabling the docker service (which starts docker-storage-setup on RHEL), which messes up the working of docker-storage-setup. Because it will then assume the device is already there. This is either a bug in docker-storage-setup or this module ;-)

Happy to be wrong here.  I made it work for me, but it was painful enough to document for others.  I'm pulling my docker package out of the official rhel-7-server-extras-rpms channel, so it's probably here to stay for a while on their end.  I don't know if upstream handles it differently or not.

This is solved by calling the docker class with:
storage_config              => false,
Which causes the /etc/sysconfig/docker-storage file to not be managed by puppet. This is not the prettiest solution, but the content of the docker-storage file ends up as desired and docker-storage-setup runs fine again.

I just wanted to chime in on this one. I'm experiencing a problem in OEL 7.1 where the order of using dm_thinpooldev => VolGroup01-ThinPool_docker and storage_driver => devicemapper does not work. The storage_driver => devicemapper actually populates the /etc/sysconfig/docker-storage config. However, the order of install.pp and config.pp seems to be the wrong way around.
What appears to happen is that puppet seems to install docker first and it launches with the default devicemapper config. It then adds the custom configuration, reloads the systemd daemon and then restarts docker. This results in a devicemapper directory in /var/lib/docker/devicemapper that does not work with the dm_thinpooldev option.
I think it has something to do with the package install portion of docker. If I set:
        class { 'docker':
            repo_opt       =>  '',
            ensure  =>  'installed',
            storage_driver =>  'devicemapper',
            dm_thinpooldev =>  'VolGroup01-ThinPool_docker',
        }

and change init.pp to this:
  class { 'docker::repos': } ->
  class { 'docker::config': } ->
  class { 'docker::install': } ~>
  class { 'docker::service': }
  contain 'docker::repos'
  contain 'docker::config'
  contain 'docker::install'
  contain 'docker::service'

I get the desired deployment effect. I think this places the docker config before installing which results the /var/lib/devicemapper folder from being created before it's launched with the correct storage configuration.
In my case I created a thin pool in my volume group using the following command:
sudo lvcreate -l100%FREE -T VolGroup01/ThinPool_docker. This type of volume does not require a separate data and metadata volume as docker manages that aspect as I believe this is the new approach for using the devicemapper drivers.

I am seeing what @lvlie  describes in Docker 1.9.1-40 and 1.10.3-44 on RHEL 7;  docker-storage-setup is checking /etc/sysconfig/docker-storage for any trace of dm.thinpooldev* and then waits for the non-existent device to be ready if it finds anything.  This seems to make it impossible to both create the thin pool and manage the /etc/sysconfig/docker-storage file simultaneously. Ultimately, my impression is that if you set both dm_* and storage_* params, you're going to have a bad time.

Bit of an old thread but stlll relevant.
CentOS packages from 1.10 seem to no longer include the storage-setup binaries(scripts).
How did you use puppet to make the thin LV? I have had a look at the puppetlabs-lvm module but they have not cut a new release with thinpool support (which I can work around), however they have no support for specifying the poolmetadata for the LV.

Several hours later I don't really see how any of this is supposed to work.
Even if I package up docker-storage-setup and install it prior to the puppet run, the result is 'docker-storage-setup' does not get run anywhere in this puppet module so the DM devices are never setup and the BEFORE line in 'docker-storage-setup.service' seems to only make RHEL7 run it during boot prior to docker.service, but this means you have to reboot after the puppet run to get the DM device.
Overall its possibly cleaner to use the LVM module or shudder do an exec, unless I am missing something I need to enable.

@eperdeme It's not as clean as it should be, but you can definitely work with thinpools on RHEL.
   class { '::docker':         service_state   => $service_state,         storage_devs    => $storage_devs,         socket_group    => $socket_group,         storage_config  => false,     }
hiera config:
docker::storage_driver:             'devicemapper' docker::storage_vg:                 'docker' docker::storage_auto_extend_pool:   'yes' docker::dm_thinpooldev:             "/dev/mapper/%{hiera('docker::storage_vg')}-docker--pool" docker::dm_fs:                      'xfs' docker::dm_use_deferred_removal:    true docker::dm_use_deferred_deletion:   true
Because docker-storage-setup is very opinionated about the storage, it's more of a burden than it should be.  But as far as this puppet module is concerned, basically have it setup /etc/sysconfig/docker-storage, docker-storage-setup.service will run, and then docker.service can start.  If you try do anything outside the scope of docker-storage-setup, you're going to end up re-creating docker-storage-setup within Puppet, which doesn't seem worthwhile as it goes against the grain.

@eperdeme I don't believe the gareth docker module creates thin pools for you. You need to configure those..
In my test vagrant environment, I did this using shell exec:

if [ ! -f ~/runonce ]; then rm -fR /etc/machine-id; systemd-machine-id-setup; (echo o; echo n; echo p; echo 1; echo ; echo; echo w) | sudo fdisk /dev/sdb; sudo pvcreate /dev/sdb1; sudo vgcreate VolGroup01 /dev/sdb1; sudo lvcreate -l100%FREE -T VolGroup01/ThinPool_docker && touch ~/runonce; fi

Yes; it's pretty dirty. I had to reset the machine-id because I'm using the same vagrant image for multiple test nodes in a cluster. But the echoes to fdisk and pv/vg/lvcreate basically format a blank drive and creates a volume group with a thinpool. I guess this would be puppetable for a greenfield deploy. The runonce file just prevents it from all running again, you can use the ** unless** or  creates directive in exec to get this same result.
In order to get it to actually use the ThinPool created, I needed to do this by calling the docker module:
class { 'docker':
    repo_opt => '',
    storage_driver => 'devicemapper',
    dm_thinpooldev => 'VolGroup00-ThinPool_docker',
}

The important part is explicitly specifying the storage_driver, otherwise it reverts to using overlay FS which is only a beta feature in the Fedora/Red Hat/CentOS kernel, and against the Red Hat recommendations for production. Had had to also set repo_opt to blank for some reason that escapes me now...  \m\e

@knackaron I tried that but the result is docker package I have seems to have no Require=docker-storage-setup in its systemd script, so the result flow is;
Docker gets install, it writes the config file for 'docker-storage-setup' and then starts the service, but as the service docker does not force 'docker-storage-setup' to run it never creates the docker-storage config file to make docker use LVM.
If you reboot prior to the service starting then it does work as https://github.com/projectatomic/docker-storage-setup/blob/master/docker-storage-setup.service has a before line for docker. Overall nothing is forcing 'docker-storage-setup' script to run prior to docker service starting. I am guessing the docker engine package you use must have a Require= line in the docker systemd script :)
Info: Applying configuration version '1479292045'
Notice: /Stage[main]/Docker::Install/Package[docker]/ensure: created
Notice: /Stage[main]/Docker::Service/File[/etc/sysconfig/docker-storage-setup]/ensure: created
Info: /Stage[main]/Docker::Service/File[/etc/sysconfig/docker-storage-setup]: Scheduling refresh of Service[docker]
Notice: /Stage[main]/Docker::Service/File[/etc/sysconfig/docker]/ensure: created
Info: /Stage[main]/Docker::Service/File[/etc/sysconfig/docker]: Scheduling refresh of Service[docker]
Notice: /Stage[main]/Docker::Service/Service[docker]/ensure: ensure changed 'stopped' to 'running'
Info: /Stage[main]/Docker::Service/Service[docker]: Unscheduling refresh on Service[docker]

@eperdeme I'm using the docker RPMs in EL7, which have unit files that establish the dependency between dss and docker.
[Unit] Description=Docker Application Container Engine Documentation=http://docs.docker.com After=network.target Wants=docker-storage-setup.service
If those aren't an option, it's simple enough to augment your unit file in /etc/systemd/system/docker.service.d.  Or you can create resources and relationships in your Puppet manifest to start dss before docker.  The second option is probably less desirable as there a number of other things that could trigger docker to start before dss is run, especially if you have a catalog application failure for some other reason.

We could definitely use some attention from @garethr around this topic, and the intent/outcome of this module in regards to dm thinpool.

alright, so I've dug into this for weeks now, including contributing updates to the devicemapper setup docs for docker.  Turns out that docker-storage-setup is in fact no longer distributed with docker, due to the default storage driver now recommended is overlay2.
Unfortunately, many RHEL family hosts don't have the ability to support even overlay in a stable fashion at the moment.
RedHat's Project Atomic also recently renamed docker-storage-setup to container-storage-setup, and I've done some fairly extensive testing of the new new service.  Bottom line, it's pretty much required if you're going to run devicemapper in direct-lvm mode.
The best thing to do at this point may be to contribute a custom type and provider to this module which provides a container_storage_setup resource or similar, in such a way that it will work with existing configs users of this module may already have deployed.
I've begun working on this, but haven't decided yet if it should be it's own puppet module, or if I should include it in this module.  I believe the rename of the tool is significant indication that it may be used for more than just docker in the near future.
The other downside here is that it's not currently distributed as a package, so it will require more puppet code to implement (hence the type and provider).


Not using GPU

I have a laptop with an Nvidia GTX 1650 and it is not using it despite having GPU processing enabled, I have downloaded the most recent drivers but it is clearly using CPU

I have the same fuckin problem and it´s really annoying, did you find the solution?

@AngusGhoul I made some important changes the latest release that should have resolved this issue for most people. What type of GPU do you have?
@diegomora02 A majority of models in UVR require at least 8GB's of V-RAM (6 being the absolute bare minimum) and it looks like you only have 4. In some cases if the application can't run on the GPU, it defaults to the CPU instead.

@Anjok07 yes I start to have this problem after I updated yesterday. I have a Geforce 3050rtx. but it´s weird because works sometimes, and sometimes not.

this one is my version Ultimate Vocal Remover v5.6.0 [2023-10-18 13:58:59]

@AngusGhoul Try these 2 things

If you're using MDX-Net, choose any segment size other than 256
Choose your GPU directly (see image). If you don't see a GPU selection that doesn't look like the image below, please reinstall the patch here

Yes, I selected the gpu manually before. I will send you screenshots of my task manager. Like I said, the program was working before the last patch. I dont know what to do but was working before :(

Sorry, i didn´t understood this. I don´t have a graphic card that uses 8 Gb. Insetad, mine is only of 2 Gb (NVidia Geforce GT 1030 2Gb) and i´m using 48 Gb of RAM memory. What are the correct configurations for using UVR with my hardware ?
Can you update UVR to it automatically detect and configure for the user´s hardware ?

I think a 1030 has a too old revision of CUDA. But even if it would work, the CUDA-cores do not have enough power to bring any performance boost compared to a decent CPU. I believe everything below GT1080 will not make sense.

@Anjok07 I have the same problem using Windows Manual Installation,
Is there a way to apply the last patch on manual installation?


need update for openwrt master version

CSS needs update for latest openwrt/luci.

@w01230 please use branch dev for luci master/latest 18.06

Hi @panda-mute , sorry for re-opening this however, the theme is great but needs updating, there is issues with the theme where things such as Memory, Total Available, Free, Buffered area is not showing any progress bar, as well as tabs not working as its being shown on one whole page.
I am currently using OpenWrt SNAPSHOT r9886 and i manually downloaded your files and placed them as per your script. So apart from the odd issues which might be .css based as the material theme updated greatly since the last version you used.

Please use the latest dev branch code.

I was using the latest version from Dev branch, however someone else made a fork and updated everything for OpenWRT 18.06.2 also from your Dev branch and corrected the issue and I decided to fork from that person and add some more code in to fix an icon issue. New to GitHub coding.


The httpc.Do method automatically adds a newline character after the request body.

Describe the bug
The httpc.Do method automatically adds a newline character after the request body.
To Reproduce
Steps to reproduce the behavior, if applicable:

The code is

client code:
package main

import (
 "bytes"
 "context"
 "encoding/json"
 "net/http"

 "github.com/zeromicro/go-zero/rest/httpc"
)

type Foo struct {
 A int `json:"a"`
 B int `json:"b"`
}

func main() {
 bs, _ := json.Marshal(Foo{A: 1, B: 2})
 http.Post("http://localhost:8080/post", "application/json", bytes.NewBuffer(bs))
 httpc.Do(context.Background(), http.MethodPost, "http://localhost:8080/post", Foo{A: 3, B: 4})
}

server code:
package main

import (
	"fmt"
	"io"

	"github.com/gin-gonic/gin"
)

func main() {
	r := gin.Default()
	r.POST("/post", func(c *gin.Context) {
		bs, err := io.ReadAll(c.Request.Body)
		if err != nil {
			c.JSON(500, err.Error())
			return
		}
		fmt.Println("req body:", string(bs))
		c.JSON(200, "ok")
	})
	r.Run(":8080")
}

The error is

Expected behavior
A clear and concise description of what you expected to happen.
Screenshots
If applicable, add screenshots to help explain your problem.
Environments (please complete the following information):

OS: [e.g. macOS]
go-zero version [1.5.3]
goctl version [e.g. 1.2.1, optional]

More description
Add any other context about the problem here.

json automatically inserts the newline character:
https://github.com/zeromicro/go-zero/blob/92e5819e9120e11361f89d2cdb0224d48327fe73/rest/httpc/requests.go#L93
https://github.com/golang/go/blob/a031f4ef83edc132d5f49382bfef491161de2476/src/encoding/json/stream.go#L201-L220

This can potentially cause an issue: If the client directly uses json.Marshal to serialize the request body for signature calculation, it will result in different signatures being computed on the client and server sides.

This can potentially cause an issue: If the client directly uses json.Marshal to serialize the request body for signature calculation, it will result in different signatures being computed on the client and server sides.

Replacing the following code with json.Marshal will also have the same problem.
https://github.com/zeromicro/go-zero/blob/f296c182f78e743af2291b564a4b6b48628547bf/rest/httpc/requests.go#L92-L93

This can potentially cause an issue: If the client directly uses json.Marshal to serialize the request body for signature calculation, it will result in different signatures being computed on the client and server sides.

Replacing the following code with json.Marshal will also have the same problem.
https://github.com/zeromicro/go-zero/blob/f296c182f78e743af2291b564a4b6b48628547bf/rest/httpc/requests.go#L92-L93

So it ultimately depends on how the user chooses. If they opt to use httpc.Do, then they must use the approach json.NewEncoder(&buf).Encode(jsonVars) to retrieve the request body's content for signature calculation.


raw_input is used, which is replaced by input in latest py version.

Windows 10
Python 3.6.1
Latest SDK
\service\samples\iothub_devicemethod_sample.py
`def iothub_devicemethod_sample_run():
try:
    ...
    print ( "Response payload         : {0}".format(response.payload) )

    raw_input("Press Enter to continue...\n")`

After changing this to input the sample worked for me

@FabianKLS
Thanks for the correction, we will update the samples in the next release.
Best Regards,
Zoltan

The samples have been update in the release:
https://github.com/Azure/azure-iot-sdk-python/releases/tag/release_2017_09_26
Best Regards,
Zoltan


🐛 [BUG]: The Example page on the official website is down

Is there an existing issue for this?

[X] I have searched the existing issues and this is a new bug.

Current Behavior
broke up
Expected Behavior
broke up
Steps To Reproduce
No response
Relevant log output
No response
Anything else?
No response

Examples should work again, please try it again (make sure you're not caching the site though 😉)


Are there plans to develop applications???

It's an awesome web project, I can't wait for that. haha

There's a work-in-progress desktop app https://github.com/excalidraw/excalidraw-desktop. No plans for mobile apps if I'm aware.

I started on a mobile app for iOS a while back, but it seems like Apple might not be allowing apps that are mostly a web app on the App Store after a preliminariy rereading of the App Store guidelines.

Progressive Web App for mobiles :)


Update readme animation

Closes https://github.com/Infleqtion/client-superstaq/issues/649

out of curiosity is there a reason we don't show the output of the last line?

i guess it would be a bit misleading because it would show the results coming back nearly instantly..

This is fine cause the passed in backend is a simulator. I always thought we don't do that cause we don't want to have too much info in the gif.

@vtomole @richrines1 just let me know if you prefer it - i can add it as a comment in the bottom line

This is a huge nit to me so I'm fine with what we have.

I always thought we don't do that cause we don't want to have too much info in the gif.

i don't think it's too much to show something like {"00": 47, "11": 53}, and imo it seems a little incomplete without it (tbh i thought my browser was hesitating or something when i first watched it)
(not necessarily related to this pr though)

@vtomole @richrines1 added the counts as a last line in the comments

Yeah of course! Also not sure where this comment should really go but the files changed makes it look like I'm making a new image (e.g., like I'm making something in Photoshop and dropping in into the folder), but I'm just updating a few lines of code in the svg file in case people want to make updates in the future


iSeqnorm pooling notebook.

I made a new pooling notebook for the iSeqnorm pipeline. I added new visualizations and wrote 4 functions that could be merged into metapool.py:
calculate_iseqnorm_pooling_volumes()
linear_transform()
read_survival()
merge_read_counts()
Unfortunately, I haven't written any tests for them.
Let me know if you need anything from me.

@RodolfoSalido The number of files to review jumped to 27. I'm not sure if this is because I inadvertently merged master into it in order to manually resolve a conflict. Looking into it.

Hi @RodolfoSalido in test_metapool.py, there are several lines calling the join() function instead of os.path.join() or defining 'from os.path import join' at the top of the file. Can you please change all occurrences to use either join() or os.path.join()? Please note that there are also a few lines that are calling string.join() which is a different function entirely so you'll want to be careful with search and replace. This should get most of the tests to pass.

Hi @RodolfoSalido it looks like there are four tests failing and one test has an error. It looks like at least some of them such as test_estimate_read_depth(), the observed results from calculate_iseqnorm_pooling_volumes() differ from the expected values of the TSV file by a single row or column. Can you please confirm that the following tests work as intended?
test_estimate_read_depth
test_find_threshold_autopool
test_merge_read_counts
test_read_pico_csv
test_read_plate_map_csv_validate_qiita_sample_names
You can review the error messages here: https://github.com/biocore/metagenomics_pooling_notebook/actions/runs/4611263674/jobs/8150742325?pr=89

@RodolfoSalido it looks like we are down to one error and two failures. I'll pull the latest and see if I can't resolve these. If I have any questions I'll ping you. Thanks!

@RodolfoSalido It looks like earlier today you pushed a commit with only a few file changes, followed by a second commit later that added most of the 27 new files that we're seeing. Can you confirm that this is correct? For the record I am usually in favor of adding as many sample input/output files as we need.

@RodolfoSalido I pushed a fix for a single test to confirm that I could push to your repo. I'll be pushing fixes for the remaining tests as soon as possible.

Yes, should be correct. The commit with the 27ish file changes included the unit_tests and a bunch of input and output files.
…

Great, thanks Rodolfo!

@RodolfoSalido All tests are passing! I will approve and merge.


joi allow produces select ignoring other cases

validating route path param with Joi.string().regex(/^[0-9a-fA-F]{24}$/).allow('me') - produces select with option me, so we cant test any other valid path params.
maybe use Joi Any.example method in this case to populate options or just leave test input and add extra info about allowed values?

Yes this is the way the Swagger UI works. If you provide .allow('me') it creates a select with no other means of input.  JOI can provide a much richer param information than the UI can deal with. I would go with text input and add some notes.


header update?

maybe this will work.I am a newbie at this. Haven't collaborated with anyone for a long time.Maybe I made a mistake
issue-#5057

Coverage decreased (-0.07%) to 88.336% when pulling 57413fe16d4d2f8dbf5691a5c4cb7c11fd671af2 on generationXcode:develop into 13a61b285c095bda7ea8e33156090ea5ccfeaef1 on moment:develop.

Hi @generationXcode,
I think current version is 2.24.0 as found here in moment.js : https://github.com/moment/moment/blob/develop/moment.js#L4555
or in the package.json : https://github.com/moment/moment/blob/develop/package.json
Also, due to the hooks.version needing to be updated, I feel that probably we could use post-hook (https://githooks.com/) to perform that task automatically ?
(Just dropping some suggestions, not owner/maintainer on this repo)

Take a look here: https://github.com/moment/moment/issues/5057#issuecomment-618592588


k3s agent should timeout when you try to join with a bad token or bad server address

Is your feature request related to a problem? Please describe.
Currently if the token isn’t valid or the server address is bad, the joining nodes keeps trying to join forever and there is no timeout
Describe the solution you'd like
Joining nodes should timeout when you try to join with a bad token or bad server address and log the cause for failure to join
Describe alternatives you've considered
Check the logs and see the interrupt the process
Feb 15 22:19:43 ip-172-31-8-63 k3s[2965]: time="2023-02-15T22:19:43Z" level=info msg="Waiting to retrieve agent configuration; server is not ready: failed to retrieve configuration from serv>

Additional info
curl -fL https://get.k3s.io | INSTALL_K3S_COMMIT=52e7a47e1afff65047a66faf13e66b20b793b64e sh -s - agent --server https://<IP>:6443 --token <BAD TOKEN>
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 27852  100 27852    0     0  53813      0 --:--:-- --:--:-- --:--:-- 53768
[INFO]  Using commit 52e7a47e1afff65047a66faf13e66b20b793b64e as release
[INFO]  Downloading hash https://k3s-ci-builds.s3.amazonaws.com/k3s-52e7a47e1afff65047a66faf13e66b20b793b64e.sha256sum
[INFO]  Downloading binary https://k3s-ci-builds.s3.amazonaws.com/k3s-52e7a47e1afff65047a66faf13e66b20b793b64e
[INFO]  Verifying binary download
[INFO]  Installing k3s to /usr/local/bin/k3s
[INFO]  Skipping installation of SELinux RPM
[INFO]  Creating /usr/local/bin/kubectl symlink to k3s
[INFO]  Creating /usr/local/bin/crictl symlink to k3s
[INFO]  Creating /usr/local/bin/ctr symlink to k3s
[INFO]  Creating killall script /usr/local/bin/k3s-killall.sh
[INFO]  Creating uninstall script /usr/local/bin/k3s-agent-uninstall.sh
[INFO]  env: Creating environment file /etc/systemd/system/k3s-agent.service.env
[INFO]  systemd: Creating service file /etc/systemd/system/k3s-agent.service
[INFO]  systemd: Enabling k3s-agent unit
Created symlink /etc/systemd/system/multi-user.target.wants/k3s-agent.service → /etc/systemd/system/k3s-agent.service.
[INFO]  systemd: Starting k3s-agent

Grooming notes: When fixing this, the dev should consider how this could affect the Rancher Manager side of things specifically the UI. So whoever fixes either talk to EM or PjM about cross-team communication.


Placeholder dots for passwords for user editing

@h-kataria wrote: When i go user edit screen, it shows blank in the password & confirm password fields(no bullets). When i try to get a password for an existing record it gives me back nil even tho i have a password saved for that user in database. I think it is because Scott made some changes a while ago to user model that does not allow us to read the password of user record in db. I am thinking that maybe we can show bullets on the screen if an existing user record is being edited as password is a required field so it has to be there and show the fields with blank if a new user record is being edited.
Related to #926

@martinpovolny seems this is a stale issue. Can you close.


feat: Provide capability to configure dns policy

…erences services by local name

Provide a description of what has been changed
Checklist

[x] Commits are signed with Developer Certificate of Origin (DCO - learn more)
[x] A PR is opened to update KEDA core (repo) (if applicable, ie. when deployment manifests are modified)
[x] README is updated with new configuration values (if applicable)

Fixes #

Thank you!

May we have a fresh release please  with this feature?

We will ship a new version of KEDA in August so I'm wondering if this is blocking you or if you are OK to wait a few week(s)?


Is this prover compatible with the Starkware onchain smart-contract ?

Can I feed the trace and proof generated by Giza into the "official" starknet verifier ?

Same question. https://github.com/starkware-libs/starkex-contracts/tree/master/evm-verifier

@maxgillett Do you know ?

It is not. One of the major differences is in the layout, with more columns present in the Giza execution trace than what is prepared by SHARP, (as Starkware is optimizing for verification time/cost over proving time). It would be interesting to list out all the changes that need to be made to make it compatible.

giza would be much more useful if it is with official Starkware verifier. Look forward to the changes to make this happen.

@xhliu, you may want to check out https://github.com/andrewmilson/sandstorm, which is focused on SHARP compatibility. Closing this issue for now.


crunchyroll -o series and season not producing expected outcome

Please follow the guide below

You will be asked some questions and requested to provide some information, please read them carefully and answer honestly
Put an x into all the boxes [ ] relevant to your issue (like that [x])
Use Preview tab to see how your issue will actually look like

Make sure you are using the latest version: run youtube-dl --version and ensure your version is 2017.01.24. If it's not read this FAQ entry and update. Issues with outdated version will be rejected.

[x] I've verified and I assure that I'm running youtube-dl 2017.01.24

Before submitting an issue make sure you have:

[x] At least skimmed through README and most notably FAQ and BUGS sections
[x] Searched the bugtracker for similar issues including closed ones

What is the purpose of your issue?

[x] Bug report (encountered problems with youtube-dl)
[ ] Site support request (request for adding support for a new site)
[ ] Feature request (request for a new functionality)
[ ] Question
[ ] Other

The following sections concretize particular purposed issues, you can erase any section (the contents between triple ---) not applicable to your issue

If the purpose of this issue is a bug report, site support request or you are not completely sure provide the full verbose output as follows:
Add -v flag to your command line you run youtube-dl with, copy the whole output and insert it here. It should look similar to one below (replace it with your log inserted between triple ```):
$ youtube-dl -v <your command line>
[debug] System config: []
[debug] User config: []
[debug] Command-line args: [u'-v', u'http://www.youtube.com/watch?v=BaW_jenozKcj']
[debug] Encodings: locale cp1251, fs mbcs, out cp866, pref cp1251
[debug] youtube-dl version 2017.01.24
[debug] Python version 2.7.11 - Windows-2003Server-5.2.3790-SP2
[debug] exe versions: ffmpeg N-75573-g1d0487f, ffprobe N-75573-g1d0487f, rtmpdump 2.4
[debug] Proxy map: {}
...
<end of log>

Description of your issue, suggested solution and other information
When downloading videos from crunchyroll and using the -o option youtube-dl replaces season with NA and replaces series with season name, incidentally it also replaces season_id and season_number with NA as well.
The full option and argument I'm passing: -o 'D:/crunchyroll/anime/%(series)s/%(season)s/%(episode_number)s - %(episode)s.%(ext)s'
A free example video that has a different series and season name: http://www.crunchyroll.com/a-certain-scientific-railgun/episode-47-silent-party-727385

http://www.crunchyroll.com/naruto-shippuden/episode-462-a-fabricated-past-696233
http://www.crunchyroll.com/konosuba-gods-blessing-on-this-wonderful-world/episode-1-give-me-deliverance-from-this-judicial-injustice-727589
http://www.crunchyroll.com/one-piece/episode-733-attack-on-a-celestial-luffys-king-kong-gun-of-anger-696989
http://www.crunchyroll.com/blue-exorcist/episode-1-small-beginnings-728285
http://www.crunchyroll.com/gintama/episode-317-the-monster-and-the-monsters-child-728627
http://www.crunchyroll.com/rwby/episode-1-the-next-step-723153
http://www.crunchyroll.com/jojos-bizarre-adventure/episode-1-jotaro-kujo-meets-josuke-higashikata-702297
http://www.crunchyroll.com/mobile-suit-gundam-iron-blooded-orphans/episode-26-new-blood-722219
http://www.crunchyroll.com/bleach/episode-317-untitled-573056


client: Fix bug where API errors weren't including the field or getting separated

Short description of the changes
This PR fixes a bug where the errors returned from the client weren't showing the impacted field and weren't getting separated when there were multiple errors associated with 1 request. This made any server-side errors a little difficult to read/fix.
I'm not super familiar with all the different forms the Details section of DetailedError can take, so if there are checks I can remove, please let me know.
How to verify that this has the expected result
Pre-requisites:

You'll need Go 1.20 and the latest Terraform installed.
You'll need a team in Honeycomb to test against.
You'll need a dataset, derived column, and SLO for that team.
You'll need an API key for that team.

Setup:

Create a new directory called providers somewhere you can easily access (but not inside the same folder as your Terraform config).
Build the provider by checking out this branch and running make build
Move the executable created to the providers directory you created earlier.mv terraform-provider-honeycombio /PATH/TO/YOUR/DIRECTORY/providers

Add the following to your ~/.terraformrc file:provider_installation {

  # Use /PATH/TO/YOUR/DIRECTORY/providers/terraform-provider-honeycombio
  # as an overridden package directory for the honeycombio/honeycombio provider.
  # This disables the version and checksum verifications for this provider and
  # forces Terraform to look for the null provider plugin in the given directory.
  dev_overrides {
    "honeycombio/honeycombio" = "/PATH/TO/YOUR/DIRECTORY/providers"
  }

  # For all other providers, install them directly from their origin provider
  # registries as normal. If you omit this, Terraform will _only_ use
  # the dev_overrides block, and so no other providers will be available.
  direct {}
}

Create another new directory (all Terraform commands will be run in this directory)
Download and save this file as main.tf in your new directory. You'll be modifying it in the steps below.
Set the following environment variables:export HONEYCOMB_API_KEY=YOUR_API_KEY_HERE
export TF_VAR_dataset=YOUR_DATASET_SLUG_HERE
export TF_VAR_slo_id=YOUR_SLO_ID_HERE

Run terraform init. You should see a yellow warning block telling you that development overrides are in place. It should look like this:

Provision resources for r/burn_alert:

Run terraform apply. This should fail with an error that look like the following:

If you check in the UI, you should see that the burn alert was not created.

Related links

API documentation

Screenshots
Before:

After:

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Comparison is base (bd9eef5) 79.24% compared to head (869d08f) 79.55%.
Report is 5 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #392      +/-   ##
==========================================
+ Coverage   79.24%   79.55%   +0.30%
==========================================
  Files          55       55
  Lines        4654     4675      +21
==========================================
+ Hits         3688     3719      +31
+ Misses        810      800      -10
  Partials      156      156

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


main

ADD ISSUE DESCRIPTION HERE
Keeps prompting( No file with "main" found in the workspace directory.)  every time I try to run code.
Version: 1.85.2
Commit: 8b3775030ed1a69b13e4f4c628c612102e30a681
User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/<IP_ADDRESS> Safari/537.36
Embedder: codespaces

\extCodespaces

It looks like this is caused by the Codespaces extension. Please file the issue in the Codespaces Discussion Forum. Make sure to check their issue reporting template and provide them relevant information such as the extension version you're using. See also our issue reporting guidelines for more information.
Happy Coding!


Add team management commands

Test for Upsun with:
# Download Upsun config
curl -s https://raw.githubusercontent.com/platformsh/cli/main/internal/config/upsun-cli.yaml > /tmp/upsun-cli.yaml

# Download the build for this PR
curl -s https://pr-1387-x5xx7sy-mpaw4nkgdwaea.eu-2.platformsh.site/platform.phar > /tmp/platform-pr-1387.phar

# Create an alias
alias upsun='CLI_CONFIG_FILE=/tmp/upsun-cli.yaml php /tmp/platform-pr-1387.phar'

# Run teams commands
upsun teams

Manually merged in 5df55fe759f50407c6ccaa060efeec4b4101c1cd


Portable comparisons (for sorting)

There are a few mixed comparisons (None with int, str with int, etc.) that do not work directly with Python 3. So add an extra element to the sorting key so that those comparisons do not occur (and work similar to Python 2).

Oh, I forgot I added some usage of six, which means this PR will conflict with #1700. Please merge it soon.

I added one more sorting fix, 184e165773b165c0235c24efa100189f902d9d7e.

Maybe that will work better.

So I've found two more comparison issues:
======================================================================
ERROR: test_multiple_unordered_lbprocs (iris.tests.unit.fileformats.pp_rules.test__all_other_rules.TestCellMethods)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/elliott/code/iris/testdir/conda/envs/env34/lib/python3.4/site-packages/Iris-1.9.0.dev0-py3.4.egg/iris/tests/unit/fileformats/pp_rules/test__all_other_rules.py", line 125, in test_multiple_unordered_lbprocs
    res = _all_other_rules(field)[CELL_METHODS_INDEX]
  File "/home/elliott/code/iris/testdir/conda/envs/env34/lib/python3.4/site-packages/Iris-1.9.0.dev0-py3.4.egg/iris/fileformats/pp_rules.py", line 881, in _all_other_rules
    coord_system=f.coord_system()),
  File "/home/elliott/code/iris/testdir/conda/envs/env34/lib/python3.4/site-packages/Iris-1.9.0.dev0-py3.4.egg/iris/coords.py", line 1346, in from_regular
    if not is_regular(coord) and count > 1:
TypeError: unorderable types: MagicMock() > int()

======================================================================
ERROR: test_multiple_unordered_rotated_lbprocs (iris.tests.unit.fileformats.pp_rules.test__all_other_rules.TestCellMethods)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/elliott/code/iris/testdir/conda/envs/env34/lib/python3.4/site-packages/Iris-1.9.0.dev0-py3.4.egg/iris/tests/unit/fileformats/pp_rules/test__all_other_rules.py", line 136, in test_multiple_unordered_rotated_lbprocs
    res = _all_other_rules(field)[CELL_METHODS_INDEX]
  File "/home/elliott/code/iris/testdir/conda/envs/env34/lib/python3.4/site-packages/Iris-1.9.0.dev0-py3.4.egg/iris/fileformats/pp_rules.py", line 881, in _all_other_rules
    coord_system=f.coord_system()),
  File "/home/elliott/code/iris/testdir/conda/envs/env34/lib/python3.4/site-packages/Iris-1.9.0.dev0-py3.4.egg/iris/coords.py", line 1346, in from_regular
    if not is_regular(coord) and count > 1:
TypeError: unorderable types: MagicMock() > int()

I'm not sure what the expected value should be for these. In those two tests, some fields of the pp mocked data are unset. @marqh Can you shed some light on these since you wrote the tests?

the tests that are failing are failing in the iris.coords
the error is due to the fact that a mock object is being evaluated compared to an integer.  this is because the tests provide a mocked version of the x_coord+name and _y_coord_name in the mock field
most mock tests only use the parts of the PP field they require
@QuLogic please may you be more specific when you say:

I'm not sure what the expected value should be for these.
which expected values? for which 'these'?

thank you
mark

The PP loader calls DimCoord.from_regular(f.bzx, f.bdx, f.lbnpt, ..., but none of these fields are set in the mocked object. Those are the fields I mean.

The PP loader calls DimCoord.from_regular(f.bzx, f.bdx, f.lbnpt, ..., but none of these fields are set in the mocked object. Those are the fields I mean.

I see.  Those metadata elements are not required for this test, the only important facet for setting the cell_methods for the lbproc codes is whether the grid is rotated or not, so I mocked out the behaviour with the _{x|y}_coord_name
I'm afraid I don't understand the nature of the test failure. I can see that the coord_system is being run on the field, but I don't see why this is now failing
we could make up a set of arbitrary values for the field to enable this test to pass, although this adds complexity to the test

Perhaps this was happening all along but on Python 2 the comparison just returns False? See below.
Python 3:
>>> from mock import MagicMock
>>> m = MagicMock()
>>> m > 1
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: unorderable types: MagicMock() > int()

Python 2:
>>> from mock import MagicMock
>>> m = MagicMock()
>>> m > 1
False

Yes, that's correct; Python 2 sorts them arbitrarily. If the actual points don't really matter, how about just setting them to 0?

Those metadata elements are not required for this test

I'm not sure that's true. The code from here onward has several conditions that depend on bdx and bdy, and are depending on mock's automatic attribute behaviour. But lbcode is set to select the right conditions earlier, so I think bdx and bdy should be set similarly.

I think bdx and bdy should be set similarly.

:+1: Yes, that makes sense. Just as we don't really care about _x_coord_name when checking cell methods, we don't care about bdx, but we need to set both just to get through that routine. (You might need to set bzx, lbnpt, bzy and lbrow as well.)
FWIW, I'd set bdx and bdy to 1. If you need to set bzx and bzy I'd set them to 0. I'd set lbnpt and lbrow to 3.

Sorry for the delay; I needed to set a few of the ones you mentioned plus x_bounds=None so it wouldn't get confused setting an inconsistent bounds.


[BUG] Use error

What happened?
A bug happened!
Version
Latest
What OS are you seeing the problem on?
Windows
What browsers are you seeing the problem on?
No response
Relevant log output
dd@wudi MINGW64 ~/work/wudi/hjcore
$ pnpm create plasmo
.../Local/pnpm/store/v3/tmp/dlx-65056    |   +2 +
Packages are hard linked from the content-addressable store to the virtual store.
  Content-addressable store is at: C:\Users\dd\AppData\Local\pnpm\store\v3
  Virtual store is at:             ../../../AppData/Local/pnpm/store/v3/tmp/dlx-65056/node_modules/.pnpm
.../Local/pnpm/store/v3/tmp/dlx-65056    | Progress: resolved 2, reused 2, downloaded 0, added 2, done
🟣 Plasmo v0.69.3
🔴 The Browser Extension Framework
🟡 Extension name: soak-legal-otter
🟡 Extension description: A basic Plasmo extension.
🟡 Author name: dd
🔵 INFO   | Creating new blank project
🔵 INFO   | Installing dependencies...
 WARN  deprecated<EMAIL_ADDRESS>Modern JS already guarantees Array#sort() is a stable sort, so this library is deprecated. See the compatibility table on MDN: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array/sort#browser_compatibility
 WARN  deprecated<EMAIL_ADDRESS>Please use @jridgewell/sourcemap-codec instead
Packages: +561
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Packages are hard linked from the content-addressable store to the virtual store.
  Content-addressable store is at: C:\Users\dd\AppData\Local\pnpm\store\v3
  Virtual store is at:             node_modules/.pnpm
Progress: resolved 627, reused 560, downloaded 0, added 561, done
<EMAIL_ADDRESS>Running install script, failed in 21.8s
<EMAIL_ADDRESS>install$ (node install/libvips && node install/dll-copy && prebuild-install) || (node install/
│ sharp: Downloading https://github.com/lovell/sharp-libvips/releases/download/v8.14.2/libvips-8.14.2-win32-x64.tar.br
│ sharp: Installation error: connect ETIMEDOUT <IP_ADDRESS>:443
│ sharp: Please see https://sharp.pixelplumbing.com/install for required dependencies
└─ Failed in 21.8s at<EMAIL_ADDRESS> ELIFECYCLE  Command failed with exit code 1.
🟠 WARN   | pnpm install exited with non-zero code: 1
🔵 INFO   | Initializing git project...
🟢 DONE   | Your extension is ready in:  C:\Users\dd\work\wudi\hjcore\soak-legal-otter

    To start hacking, run:

       cd soak-legal-otter
       pnpm dev

    Visit https://docs.plasmo.com for documentation and more examples.

(OPTIONAL) Contact Details
<EMAIL_ADDRESS>Code of Conduct

[X] I agree to follow this project's Code of Conduct
[X] I checked the current issues for duplicate problems.

@WuChenDi can you try again, it seems your internet timed out when trying to install sharp. It was upgraded recently, so it's very likely your npm proxy has yet to update its cache.


be197cf580d causes corruptions at speed=0 or 1

https://github.com/xiph/rav1e/commit/be197cf580d5e45d867d09d75d6bb7b2da2c77c9

Fixed in #375.


contol plane NIC requirements

Describe the bug
The current requirement is that the control plane has 3 NICs and 1 InfiniBand connection
NIC1: public internet
NIC2: idrac
NIC3: PXE
Infiniband: ?
the control plane should only require 2 NICs:
NIC1: public internet
NIC2: internal communication + sidebanded idrac/pxe
Should also include an option to add additional NICs if using dedicated idrac/pxe network
We need to remove the requirement for IB since we are not using it in this situation.

@j0hnL The resource pool nodes can get away with a sideband idrac/eth, but since the control plane needs to be able to offer DHCP and DNS for iDRAC as well as host, then there should be 3 Ethernet connections on the control plane:

A NIC for access to the outside world (this may not even be necessary if every host in your datacenter is connected to the internet, e.g., flat topology)
A NIC for managing the out-of-bound network (iDRAC, IPMI, etc.).
A NIC for managing the in-bound network (PXE, host-to-host communication, Ansible, SSH, etc.)

Infiniband connections would be separate from this.

@j0hnL and @lwilson, Control Plane configures the management network for the devices under ToR, so there should be one connection to from control plane to pass through, e.g. NIC1
Network connection to outside world, e.g. NIC2 and Network connection for PXE (HPC communication) e.g. NIC3 -- We can combine these 2.
If you want NIC1, NIC2, NIC3 can be combined but then management and host network will be on the same path which will trigger security issues, I would not recommend this.
Network Connection to IB Switch e.g. NIC4 (This is optional, if user does not have IB switch this can be skipped.)

Solved in devel branch in Issue https://github.com/dellhpc/omnia/pull/834