introduce dummy time functions for testing retrying

Right now two test cases for api_callable are marked as 'skip' because of the lack of mocking time-related features.  We should enable them.

This has been around for 3 years, and I kinda doubt we're going to fix it any time soon.  Closing out.


Add Display name (if !== URL) for a server in the kernel picker

Fixes #11381

[x] Pull request represents a single change (i.e. not fixing disparate/unrelated things in a single PR).
[x] Title summarizes what is changing.
[x] Has a news entry file (remember to thank yourself!).
[ ] Appropriate comments and documentation strings in the code.
[ ] Has sufficient logging.
[ ] Has telemetry for feature-requests.
[ ] Unit tests & system/integration tests are added/updated.
[ ] Test plan is updated as appropriate.
[ ] package-lock.json has been regenerated by running npm install (if dependencies have changed).

Instead of (Remote) it now looks like this if you specify a display name (AzureML as example)

Codecov Report

Merging #11457 (00ec3ab) into main (06ca869) will increase coverage by 0%.
The diff coverage is 93%.

@@          Coverage Diff           @@
##            main   #11457   +/-   ##
======================================
  Coverage     63%      63%
======================================
  Files        477      477
  Lines      34154    34181   +27
  Branches    5544     5547    +3
======================================
+ Hits       21591    21611   +20
+ Misses     10506    10502    -4
- Partials    2057     2068   +11

Impacted Files
Coverage Δ

src/kernels/jupyter/launcher/serverUriStorage.ts
64% <ø> (ø)

...rc/notebooks/controllers/controllerRegistration.ts
86% <ø> (-1%)
:arrow_down:

.../notebooks/controllers/vscodeNotebookController.ts
82% <92%> (+<1%)
:arrow_up:

src/platform/common/utils/localize.ts
76% <100%> (+<1%)
:arrow_up:

...ve-window/debugger/jupyter/debugCellControllers.ts
82% <0%> (-14%)
:arrow_down:

src/kernels/execution/notebookUpdater.ts
88% <0%> (-8%)
:arrow_down:

src/platform/common/cancellation.ts
59% <0%> (-4%)
:arrow_down:

src/kernels/jupyter/jupyterKernelService.node.ts
74% <0%> (-2%)
:arrow_down:

src/kernels/installer/productInstaller.node.ts
78% <0%> (-2%)
:arrow_down:

src/kernels/raw/launcher/kernelProcess.node.ts
72% <0%> (-2%)
:arrow_down:

... and 8 more


🐛 Emoji is breaking

Description
Emoji css is breaking
I'm a GSSoC'23 contributor please assign me this issue under gssoc label and level.
Steps to reproduce
go to website click on post
Screenshots

Desktop (please complete the following information)
No response
Smartphone (please complete the following information)
No response
Additional information
No response
Record

[X] I agree to follow this project's Code of Conduct
[X] I have checked the existing issues
[X] I'm a GSSoC'23 contributor
[X] I want to work on this issue

Description
Emoji css is breaking I'm a GSSoC'23 contributor please assign me this issue under gssoc label and level.
Steps to reproduce
go to website click on post
Screenshots

### Desktop (please complete the following information)
_No response_
Smartphone (please complete the following information)
No response
Additional information
No response
Record

[x]  I agree to follow this project's Code of Conduct
[x]  I have checked the existing issues
[x]  I'm a GSSoC'23 contributor
[x]  I want to work on this issue

sure, you can work on this! :)


Consolidate Packing

Currently, different packing algorithms assume different things and follow different semantics for the user-set configs. Because of it, there exist some code duplicates and it's not friendly for new packing algorithms. This PR is an attempt to address this issue and consolidate the three existing packing algorithms.
Major behavior changes are:

FirstFitDecreasingPacking and ResourceCompliantRoundRobinPacking now take container-level resource constraints from topology.container.cpu, topology.container.ram, and topology.container.disk instead of from hints.
FirstFitDecreasingPacking and ResourceCompliantRoundRobinPacking now determine padding resource by taking max(containerResource * padding%, paddingAmount)
More details can be found in AbstractPacking class

This PR also revamp tests and code reuse. Changed packing tests to all extend from CommonPackingTest and uses helper methods for each test scenarios. Assertion expectations are changed accordingly due to the behavior changes introduced by the new determination of padding resource amount.

Overall LGTM.
Any tests done? like verifying physical plan with --dry-run?

@nwangtw Yes. Verified by running modified word_count topologies with all 3 packing algorithms in dry-run mode.


Spotify redirection works but doesn't login

Hi, I have cloned the SpotifyAPIExample but edit some features, and now I have a problem, when I first login, everything worked, but after logout using this Button(action: spotify.api.authorizationManager.deauthorize) it doesn't work anymore.
When I click login, I'm redirected to Safari, I can login to Spotify, I can redirect to the app but nothing happen, it just keep saying that I'm not logged in.
Thanks for your help

Did you resolve this issue? If not, please make the source code of your app available in order for me to investigate.


chore(core/layers): align info method of trait Access and trait LayeredAccess

Which issue does this PR close?
No
Rationale for this change
The metadata method of LayeredAccess and info method of LayeredAccess is misaligned.
we used to use metadata and changed it to info.
What changes are included in this PR?

rename metadata method of trait LayeredAccess:
fn metadata(&self) -> Arc<AccessorInfo> => fn info(&self) -> Arc<AccessorInfo>

Are there any user-facing changes?
LayeredAccess::metadata => LayeredAccess::info

This is a breaking change, we will need to wait for https://github.com/apache/opendal/discussions/5257


Float64 support for cap-mbps parameter

I've made changes for cap-mbps to support float64, particularly for scenarios where bandwidth is severely limited and there's a necessity to limit bandwidth to a fraction of 1 mbps.

This PR replaces https://github.com/Azure/azure-storage-azcopy/pull/919

Looks good to me.

Thanks @9137


[Feature request] Add code highlighting for String interpolation in Scala

I would like to know if it is possible to add highlighting of code that is part of Scala's string interpolation:
  val data = lines.map(_.toInt).sliding(2).toList
  println(s"We have ${data.size} pairs")

The part between ${ and } in a string that starts with s" should be highlighted as if it is actual code.

Test drive:

IntelliJ:

Yes, it's possible. I'll add it next week.

That sounds great. Do you know the Scala syntax for string interpolation? The string must be prefixed wits s or f, and then can contain $var and ${expression} parts (plus a formatting suffix in case of an f prefix). More info on https://docs.scala-lang.org/overviews/core/string-interpolation.html


Remove the ResultSet object

This commit removes the ResultSet object but also makes PreparedStatement objects quack the same as ResultSet.

I'm not sure about this PR.  It seems like we shouldn't need the ResultSet object, but this patch impacts the behavior of prepare when someone passes :results_as_hash => true to the database.  This PR breaks ActiveRecord, so I'm going to port the code ActiveRecord is calling to a SQLite3 test so we can prevent regressions.
FWIW, Rails is defaulting to :results_as_hash => true, and I don't think that's a good idea.  I will also try to change Rails to not do that (if possible).

I'm going to close this. We can revisit it in the future, but I don't think it's necessary for 2.0.0.

I wrote my comment before I read yours. I'm fine with closing it.

But as an aside, we should probably add an integration test for litestack to the CI suite. Unfortunately it looks broken right now.

💯


Demo crashes when rendering native numbers view in modal

We noticed this issue because we'd like to access the TurboNavigationController inside of a native modal in order to invoke some navigation from there.
To reproduce the issue you simply need to add "presentation": "modal", in line 21 of path-configuration.json: https://github.com/hotwired/turbo-ios/blob/2277f7dcd52c3f8eac735980166e64bd5745272b/Demo/path-configuration.json#L21
After that, when you run the demo, you can click on "intercept with a native view" and click on one of the rows. Now the app will crash.
It seems as if the navigationController in this case is simply an instance of UINavigationController and not of TurboNavigationController, so the forced cast lets the app crash. I think this is caused by this line here: https://github.com/hotwired/turbo-ios/blob/2277f7dcd52c3f8eac735980166e64bd5745272b/Demo/Navigation/TurboNavigationController.swift#L89
So the questions are:

Is this a bug, or intended behavior?
If it's intended, how do we achieve navigation from within a native modal instead?

N. b. I tested the same thing with the android demo where it doesn't crash, so I suspect this is indeed a bug.

I'd assume this is a bug, you should definitely be able to present native screens modally.
Can you attach a stack trace or the exception name and line number? That might help debug.

Sure!
The culprit is this line: https://github.com/hotwired/turbo-ios/blob/2277f7dcd52c3f8eac735980166e64bd5745272b/Demo/NumbersViewController.swift#L34
The problem is that the viewController that gets rendered in the modal is wrapped inside a UINavigationController which cannot be cast to TurboNavigationController. It crashes with an Could not cast value of type 'UINavigationController' in the aforementioned line.
I went ahead an fixed this issue but my solution is probably not good... I could provide it as an pull request if you'd like to see the code.

Huh, are you sure you haven't changed other code, too? If I make the single change to the Path Configuration you outlined I can present the native view as a modal without issue.
diff --git a/Demo/path-configuration.json b/Demo/path-configuration.json
index a52bce9..38c3966 100644
--- a/Demo/path-configuration.json
+++ b/Demo/path-configuration.json
@@ -18,7 +18,8 @@
         "/numbers$"
       ],
       "properties": {
-        "view-controller": "numbers"
+        "view-controller": "numbers",
+        "presentation": "modal"
       }
     },
     {

You just missed the step of clicking on one of the rows. This also threw me off at first :-)
If you don't click, the view does not have to initiate navigation and hence does not access the non cast-able navigation controller.


Chapter 13. Model with Embedding layers outputs NaN

Hi,
I tried to implement this model as stated in book - however during training I have NaNs everywhere.
I suspect that the weights got to Inf or input_dim is wrong dimension (those are solutions from SO) but I'm not sure why.
Here is my code to replicate
# Prepare data
housing_file_path = 'https://raw.githubusercontent.com/ageron/handson-ml/master/datasets/housing/housing.csv'
housing_df = pd.read_csv(housing_file_path)
X_train_regular = housing_df.drop(columns=['median_house_value', 'ocean_proximity']).to_numpy()
X_train_categorical = housing_df['ocean_proximity'].to_numpy()
y_train = housing_df['median_house_value'].to_numpy()

NUM_OOV_BUCKETS = 2
VOCAB = ['NEAR BAY', '<1H OCEAN', 'INLAND', 'NEAR OCEAN', 'ISLAND']

indices = tf.range(len(VOCAB), dtype=tf.int64)
table_init = tf.lookup.KeyValueTensorInitializer(VOCAB, indices)
table = tf.lookup.StaticVocabularyTable(table_init, NUM_OOV_BUCKETS)

regular_input = keras.layers.Input(shape=[8])
categorical_input = keras.layers.Input(shape=[], dtype=tf.string)
cat_indices = keras.layers.Lambda(lambda cats: table.lookup(cats))(categorical_input)
cat_embed = keras.layers.Embedding(input_dim=len(VOCAB) + NUM_OOV_BUCKETS, output_dim=2)(cat_indices)
concat_inputs = keras.layers.concatenate([regular_input, cat_embed])
#Dense1 = keras.layers.Dense(10, 'relu')(concat_inputs)
outputs = keras.layers.Dense(1)(concat_inputs)

model = keras.models.Model(inputs=[regular_input, categorical_input],
                           outputs=[outputs])

model.compile(optimizer='adam', loss='mean_squared_error', metrics=keras.metrics.RootMeanSquaredError())

history = model.fit((X_train_regular, X_train_categorical), y_train, batch_size=32, epochs=10, validation_split=0.1, use_multiprocessing=True)

Result:
Epoch 1/10
581/581 [==============================] - 1s 1ms/step - loss: nan - root_mean_squared_error: nan - val_loss: nan - val_root_mean_squared_error: nan
Epoch 2/10
581/581 [==============================] - 1s 1ms/step - loss: nan - root_mean_squared_error: nan - val_loss: nan - val_root_mean_squared_error: nan
Epoch 3/10
581/581 [==============================] - 1s 1ms/step - loss: nan - root_mean_squared_error: nan - val_loss: nan - val_root_mean_squared_error: nan
Epoch 4/10
581/581 [==============================] - 1s 1ms/step - loss: nan - root_mean_squared_error: nan - val_loss: nan - val_root_mean_squared_error: nan
Epoch 5/10
581/581 [==============================] - 1s 1ms/step - loss: nan - root_mean_squared_error: nan - val_loss: nan - val_root_mean_squared_error: nan
Epoch 6/10
581/581 [==============================] - 1s 1ms/step - loss: nan - root_mean_squared_error: nan - val_loss: nan - val_root_mean_squared_error: nan
Epoch 7/10
581/581 [==============================] - 1s 1ms/step - loss: nan - root_mean_squared_error: nan - val_loss: nan - val_root_mean_squared_error: nan
Epoch 8/10
581/581 [==============================] - 1s 1ms/step - loss: nan - root_mean_squared_error: nan - val_loss: nan - val_root_mean_squared_error: nan
Epoch 9/10
581/581 [==============================] - 1s 1ms/step - loss: nan - root_mean_squared_error: nan - val_loss: nan - val_root_mean_squared_error: nan
Epoch 10/10
581/581 [==============================] - 1s 1ms/step - loss: nan - root_mean_squared_error: nan - val_loss: nan - val_root_mean_squared_error: nan

Hi @mlkonopelski ,
Thanks for your question. I've copied your code in the following Colab notebook, to debug it:
https://colab.research.google.com/drive/1DMeCPC93ot417NNqpCBgEkA4UpEwik1v?usp=sharing
I just called dropna() on housing_df, and now there's no nan during training.
Indeed, the housing dataset contains a few nan values, and if you don't fix them before training, then things go wrong. That's because any computation involving nan returns nan. So as soon as one nan value is encountered, the model weights get updated to nan, and after that there's no way to get any other result.
The other thing you should do is to scale the numerical features. Please see the chapter 2 for more details.
Lastly, there's now a second edition of my book, based on TensorFlow 2 instead of 1. It's much simpler to use. The code examples are open source, you can check them out at https://github.com/ageron/handson-ml2
Hope this helps.


GSD Request

--- GSD JSON ---
{
  "vendor_name": "Linux",
  "product_name": "Kernel",
  "product_version": "versions from v5.4 to before v5.10.83",
  "vulnerability_type": "unspecified",
  "affected_component": "unspecified",
  "attack_vector": "unspecified",
  "impact": "unspecified",
  "credit": "",
  "references": [
    "https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id=28b1987ef5064dd5c43538ba1168ef7b801f3cad",
    "https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id=90d0736876c50ecde1a3275636a06b9ddb1cace9"
  ],
  "extended_references": [
    {
      "type": "commit",
      "value": "28b1987ef5064dd5c43538ba1168ef7b801f3cad",
      "note": "introduced"
    },
    {
      "type": "commit",
      "value": "90d0736876c50ecde1a3275636a06b9ddb1cace9",
      "note": "fixed"
    }
  ],
  "reporter": "joshbressers",
  "reporter_id": 1692786,
  "notes": "",
  "description": "mlxsw: spectrum: Protect driver from buggy firmware\n\nThis is an automated ID intended to aid in discovery of potential security vulnerabilities. The actual impact and attack plausibility have not yet been proven.\nThis ID is fixed in Linux Kernel version v5.10.83 by commit 90d0736876c50ecde1a3275636a06b9ddb1cace9, it was introduced in version v5.4 by commit 28b1987ef5064dd5c43538ba1168ef7b801f3cad. For more details please see the references link."
}
--- GSD JSON ---

This issue has been assigned GSD-2021-1002475


Camera Permission issue with Detox Android

Permission Issue with Android : I have this issue on Detox 18.13.0, React Native 0.63.4, API 30 for my Android emulator. It works perfectly for camera permission on iOS but on Android, whatever I set NO or YES on permissions, they are always enabled.
If I change theses permissions on the app before launching detox test, all permissions are enabled again.
@noomorph @d4vidi

@d4vidi, do we have those camera permissions correctly implemented for Android?

beforeAll(async () => {
await device.launchApp({newInstance: true, permissions: {camera: ‘NO’}});
await device.launchApp({ permissions : { location: ‘always’ }})
await device.launchApp({ permissions : { notifications: ‘YES’ }})
});
So here camera permissions is not taking NO permission, always camera is shown in ON state when detox test is run
@noomorph

Hey @rajithaswoop, unfortunately, the permissions argument to launchApp() is only applicable on iOS, at the moment. You are welcomed to follow this issue for support on Android, in the future. We also accept contributions 😄


SILog NaN for higher batch size

Hi, so in my case is that i want to use multiple GPU (4 GPU) using batch size of 8, it works well.
but then i see that each GPU only utilized half of its capacity, then i tried to increase my batch size to 12.
but the same error always reappear not OOM but "SILog is NAN stopping training".
do you know why this is happening? or has anyone encounter similar problem?

Ive found the root cause to this problem. so at some part of the label images, all the pixel is 0 (meaning that the Log Loss would always be 0). eliminating these labels fixed this issue.
tl;dr
caused by uncleaned dataset


[FRONTEND][Keras] Add support for tf.Keras networks in Relay Keras frontend

Add support for tf.Keras networks in Relay Keras frontend:

Make Relay Keras frontend support networks created using Tensorflow (1.13) Keras implementation (tf.Keras)
Modify Keras frontend tests to run from a class rather than a function based script
Adjust Keras frontend tests to run with both 'Keras' and 'tf.Keras'
Change "TestKeras.test_forward_merge" to validate instances by class name rather than instance type

@tqchen @apivovarov @jwfromm can you have a look on this PR?

Thanks for making this change, supporting tf.keras is definitely valuable and actually also gives us the ability to import TF2.0 models through a saved Keras model. LGTM!

Thanks @jwfromm @leandron !


(DOCS) PUP-9576 clarify catalog find behavior

Clarifying what the catalog find command retrieves.

CLA signed by all contributors.


nuke: don't power-off machines when not rebooting

This ensures jobs that time out can still have their logs gathered.
Signed-off-by: Josh Durgin<EMAIL_ADDRESS>

@susebot test deploy

duplicate of #1149

it's similar, in this case the intended user is the supervisor via kill_job(save_logs=True) -> https://github.com/ceph/teuthology/blob/master/teuthology/kill.py#L239-L240

merging for now to fix the issues in the lab, we can change it later if desired

This PR merge appears to correspond exactly to when we started seeing errors like
2021-03-19T08:40:32.182 DEBUG:teuthology.orchestra.run.smithi041:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip --
2021-03-19T08:40:32.191 DEBUG:teuthology.orchestra.run.smithi060:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip --
2021-03-19T08:40:32.916 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cluster2-client.mirror.49320.log.gz already exists;  not overwritten
2021-03-19T08:40:32.953 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cluster1-client.mirror.38872.log.gz already exists;  not overwritten
2021-03-19T08:40:32.962 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cluster2-client.mirror.3.38872.log.gz already exists;        not overwritten
2021-03-19T08:40:33.607 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cluster1-client.mirror.40712.log.gz already exists;  not overwritten
2021-03-19T08:40:33.607 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cluster1-client.mirror.50723.log.gz already exists;  not overwritten
2021-03-19T08:40:33.608 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cluster1-client.mirror.3.38726.log.gz already exists;        not overwritten
2021-03-19T08:40:33.613 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cluster2-client.mirror.2.40712.log.gz already exists;        not overwritten
2021-03-19T08:40:33.614 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cluster2-client.mirror.1.38808.log.gz already exists;        not overwritten
2021-03-19T08:40:33.614 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cluster2-client.mirror.38726.log.gz already exists;  not overwritten
2021-03-19T08:40:33.616 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cluster2-client.mirror.1.50723.log.gz already exists;        not overwritten
2021-03-19T08:40:33.622 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cluster2.audit.log.gz already exists;        not overwritten
2021-03-19T08:40:33.703 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cluster1-client.mirror.1.38638.log.gz already exists;        not overwritten
2021-03-19T08:40:33.703 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cluster2.log.gz already exists;      not overwritten
2021-03-19T08:40:33.703 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cluster2-client.mirror.0.38771.log.gz already exists;        not overwritten
2021-03-19T08:40:33.709 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cluster1-client.mirror.2.49320.log.gz already exists;        not overwritten
2021-03-19T08:40:33.713 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cluster1-client.mirror.2.38678.log.gz already exists;        not overwritten

I think we should revert until this is resolved

This PR merge appears to correspond exactly to when we started seeing errors like
2021-03-19T08:40:32.182 DEBUG:teuthology.orchestra.run.smithi041:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip --
2021-03-19T08:40:32.191 DEBUG:teuthology.orchestra.run.smithi060:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip --
2021-03-19T08:40:32.916 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cluster2-client.mirror.49320.log.gz already exists;  not overwritten
2021-03-19T08:40:32.953 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cluster1-client.mirror.38872.log.gz already exists;  not overwritten
2021-03-19T08:40:32.962 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cluster2-client.mirror.3.38872.log.gz already exists;        not overwritten
2021-03-19T08:40:33.607 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cluster1-client.mirror.40712.log.gz already exists;  not overwritten
2021-03-19T08:40:33.607 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cluster1-client.mirror.50723.log.gz already exists;  not overwritten
2021-03-19T08:40:33.608 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cluster1-client.mirror.3.38726.log.gz already exists;        not overwritten
2021-03-19T08:40:33.613 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cluster2-client.mirror.2.40712.log.gz already exists;        not overwritten
2021-03-19T08:40:33.614 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cluster2-client.mirror.1.38808.log.gz already exists;        not overwritten
2021-03-19T08:40:33.614 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cluster2-client.mirror.38726.log.gz already exists;  not overwritten
2021-03-19T08:40:33.616 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cluster2-client.mirror.1.50723.log.gz already exists;        not overwritten
2021-03-19T08:40:33.622 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cluster2.audit.log.gz already exists;        not overwritten
2021-03-19T08:40:33.703 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cluster1-client.mirror.1.38638.log.gz already exists;        not overwritten
2021-03-19T08:40:33.703 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cluster2.log.gz already exists;      not overwritten
2021-03-19T08:40:33.703 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cluster2-client.mirror.0.38771.log.gz already exists;        not overwritten
2021-03-19T08:40:33.709 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cluster1-client.mirror.2.49320.log.gz already exists;        not overwritten
2021-03-19T08:40:33.713 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cluster1-client.mirror.2.38678.log.gz already exists;        not overwritten

I think we should revert until this is resolved

this issue has been seen with teuthology_sha1: b96569170f15eae4604f361990ea65737b28dff1, which was before this merged
https://pulpito.ceph.com/yuriw-2021-03-19_00:20:13-rados-wip-yuri-pacific_3.18.21-distro-basic-smithi/5979157/
2021-03-19T05:51:46.619 INFO:teuthology.task.internal.syslog:Compressing syslogs...
2021-03-19T05:51:46.619 DEBUG:teuthology.orchestra.run.smithi097:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip --
2021-03-19T05:51:46.641 DEBUG:teuthology.orchestra.run.smithi205:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip --
2021-03-19T05:51:46.686 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;   not overwritten
2021-03-19T05:51:46.687 DEBUG:teuthology.orchestra.run:got remote process result: 123
2021-03-19T05:51:46.688 ERROR:teuthology.run_tasks:Manager failed: internal.syslog


The NaN problem in Iteration Reconstruction

Hi
When I used the IR like SART in LEAP, I tried to use the noise which from the diffusion model, as the initial input and the SART work. But when I printed the output, I found there are many NaN (Not all,but most ) in the results.That was very strange.
I would like to know are there any limitations on the IR input? And why would the results will work out NaN results?
Best,

And here is the output, where x_next_2 is the input of IR:
x_next_2:tensor([[[[ -95.6140,  116.9636,  131.4474,  ..., -109.2653, -243.9316,
149.8053],
[-389.6929,  -13.0061,  401.6941,  ..., -143.1181, -112.7916,
-189.2075],
[ 123.8581,  235.6588, -209.9771,  ...,  260.4404, -272.1484,
-56.4938],
...,
[ -62.2912,  105.1640,   54.2454,  ..., -116.4751,   16.5664,
292.3446],
[ -39.8778,  150.4883,  193.5163,  ...,  -50.5344, -190.6261,
416.4268],
[-711.3794,   -9.8581,  156.0905,  ..., -438.4397, -486.9677,
101.1560]]]], device='cuda:0', dtype=torch.float64)
x_next2_shape:torch.Size([1, 1, 512, 512])
x_next2 max:1210.9393616716463
x_next2 min:-1165.4740936310213
proj_shape:torch.Size([1000, 1, 900])
SART iteration 1 of 10
SART iteration 2 of 10
SART iteration 3 of 10
SART iteration 4 of 10
SART iteration 5 of 10
SART iteration 6 of 10
SART iteration 7 of 10
SART iteration 8 of 10
SART iteration 9 of 10
SART iteration 10 of 10
x_nextIR:tensor([[[  0.0000,      nan,      nan,  ...,      nan,      nan, 149.8053],
[     nan,      nan,      nan,  ...,      nan,      nan,      nan],
[     nan,      nan,      nan,  ...,      nan,      nan,      nan],
...,
[     nan,      nan,      nan,  ...,      nan,      nan,      nan],
[     nan,      nan,      nan,  ...,      nan,      nan,      nan],
[     nan,      nan,      nan,  ...,      nan,      nan,      nan]]],
device='cuda:0', dtype=torch.float64)
x_nextIR_shape:torch.Size([1, 512, 512])
x_nextIR max:nan
x_nextIR min:nan

It seems like your inputs are doubles, but LEAP requires 32-bit floats.  What version of LEAP are you using?  The newest version, v1.15, should check your data type and return an error if they are invalid (i.e., not float32, wrong shape, etc).
I'm not sure what kind of input your are giving SART, but tomographic iterative reconstruction algorithms do not perform well when they are seeded with input that have high frequency components.  Some of these high frequency error will never go away and others will take an enormous amount of iterations to suppress.

Are you still having issues once you switched to using 32-bit floats?

I'm so sorry for not replying to your message earlier, I have tried to change the data to 32-bit float, and the problems are solved. Thanks for your help!


[PT Run > Calc plugin] Error message, when calculation not possible

📝 Provide a description of the new feature
If the user enters a calculation which can't be calculated for example because of an overflow, we should provide an error message for the user. The windows calculator does this.
Calculation possible:<PHONE_NUMBER>116151815118*9999999999
Calculation not possible:<PHONE_NUMBER>116151815118*100000000001
Screenshots

("Überlauf" means in English "overflow".)

If you'd like to see this feature implemented, add a 👍 reaction to this post.

this could be handled by https://github.com/microsoft/PowerToys/issues/2265

also this would be hard to capture as the current engine either has a successful calculation or nothing


Question - OpenThread - Sending "multicast" UDP messages and setting retransmits

Hello there,
I am using openthread CLI example on Nordic nRF52840 devkits to demo mesh networking.
Using the openthread/udp lib to send periodic multicast messages ( every 5s from one of the devices), and sending ACKs back from the devices that receive messages straight back to the sender of the original message, by storing the "peer address" of the sender which i get from the original message and sending the ACK message back to that address.
I have reduced the gain and spread my devices out so that I can see the "Mesh" actually working - meaning messages should propagate by hopping to adjacent nodes. They're all FTD.
I am noticing a 20% message loss in some nodes farther away from the sender but within reach of other FTD nodes. I am suspecting the retransmits/hops aren't enabled or not set right.
Questions:
Does the underlying openthread link layer has hops/retransmits built in?
If so, how do i control the number of retransmits (upon reception of a multicast message) of each node ? There are some settings around "Direct" vs "indirect" msg retransmits. I am not using indirect (polling). Direct retransmit is set to 3, does that mean each node that receives a message will in turn retransmit the message 3x as well? or is this only applied to the original sender of a message and not to other nodes?
If not, does going to upper layers such as Coap provide the true meshing and retransmit features?

FTDs always support mesh routing as that is required by the Thread protocol specification.
OpenThread currently implements 3 MAC retransmissions, as specified in Thread 1.x. However, in practice, we have found a value of 15 to be more appropriate. You can configure this value with the OPENTHREAD_CONFIG_MAC_DEFAULT_MAX_FRAME_RETRIES_DIRECT build configuration.
If you require end-to-end reliability, then transport/application-layer reliability mechanisms are also strongly encouraged.
Hope that helps.

Thanks Jonathan for the quick reply.
OK, does 15 retries apply to the original message sender MAC only or to the all the nodes that receive the message too?
e.g. say i have 3 nodes, A, B and C. A only sees B. C only sees B. And B sees both A and C.
Questions:
If A sends a message, it will retry sending it R times (determined by the OPENTHREAD_CONFIG_MAC_DEFAULT_MAX_FRAME_RETRIES_DIRECT), is that correct? Or does it get an ACK at the MAC layer from other devices and doesn't retry that many times?
Does B also retry sending the message R times upon receiving it?
Cranking R to 15, even with a handful of nodes (eg 4) could lead to 60 msg retries, do you think RF transmit over-running each other could happen? or are there systems in place in openthread to avoid that?

OK, does 15 retries apply to the original message sender MAC only or to the all the nodes that receive the message too?
e.g. say i have 3 nodes, A, B and C. A only sees B. C only sees B. And B sees both A and C.

Since the retry value is at the MAC layer, only the nodes that have implemented 15 retries will retransmit 15 retries.

Questions:
If A sends a message, it will retry sending it R times (determined by the OPENTHREAD_CONFIG_MAC_DEFAULT_MAX_FRAME_RETRIES_DIRECT), is that correct? Or does it get an ACK at the MAC layer from other devices and doesn't retry that many times?

A will retransmit until it receives an ACK from the intended receiver or it has already tried OPENTHREAD_CONFIG_MAC_DEFAULT_MAX_FRAME_RETRIES_DIRECT times.

Does B also retry sending the message R times upon receiving it?

B will also retry sending the message OPENTHREAD_CONFIG_MAC_DEFAULT_MAX_FRAME_RETRIES_DIRECT times, whatever that is set to on B.

Cranking R to 15, even with a handful of nodes (eg 4) could lead to 60 msg retries, do you think RF transmit over-running each other could happen? or are there systems in place in openthread to avoid that?

In a very good RF environment (i.e. with no message loss), you will only see 1 transmission per device. In a very poor RF environment, you could certainly see up to 15 retries per device. When you have a poor RF environment you basically have to choose between retransmitting many times or not have connectivity.
Hope that helps.

Very helpful, thanks Jonathan.
In the scenario of a multicast UDP message, do you know what the condition(s) is/are for retries to stop? Is it an ACK from adjacent node? i.e. in the example above does node "A" stop as soon as it hears and ACK from B, relying on B to propagate the message to C? Or does it retry up to OPENTHREAD_CONFIG_MAC_DEFAULT_MAX_FRAME_RETRIES_DIRECT  to hear from C also?
side note: looks like with otLinkGetTxDirectRetrySuccessHistogram() i can get insight into how well the retransmits are working.

In the scenario of a multicast UDP message, do you know what the condition(s) is/are for retries to stop? Is it an ACK from adjacent node? i.e. in the example above does node "A" stop as soon as it hears an ACK from B, relying on B to propagate the message to C? Or does it retry up to OPENTHREAD_CONFIG_MAC_DEFAULT_MAX_FRAME_RETRIES_DIRECT to hear from C also?

IPv6 multicast translates to IEEE 802.15.4 broadcast. In the broadcast case, ACKs are not sent in response. Current default is a single transmission per broadcast message. However, you can experiment with larger values using OPENTHREAD_CONFIG_MAC_TX_NUM_BCAST.

side note: looks like with otLinkGetTxDirectRetrySuccessHistogram() i can get insight into how well the retransmits are working.

Yes.

IPv6 multicast translates to IEEE 802.15.4 broadcast. In the broadcast case, ACKs are not sent in response.

Does this mean that a IEEE 802.15.4 broadcast is not taking advantage of the mesh feature? i.e. messages won't propagate through the network?
Is there a better alternative layer to use for what I am trying to do that includes ACKing to ensure messages are received? All i need to achieve is sending messages from any node to all others. Any node can send the message, and they're all FTD. and i need the Mesh feature to work (hence why I am using OpenThread).

Does this mean that a IEEE 802.15.4 broadcast is not taking advantage of the mesh feature? i.e. messages won't propagate through the network?

Link-local IPv6 multicast translates to a single IEEE 802.15.4 broadcast.
Non-link-local scope IPv6 multicast (e.g. realm-local or larger scope) is forwarded using RFC 7731, which operates at the network layer, not link layer. MPL adds its own redundancy by broadcasting each message 2-3 times on each router. You can try using larger values by changing kMplRouterDataMessageTimerExpirations.

Is there a better alternative layer to use for what I am trying to do that includes ACKing to ensure messages are received? All i need to achieve is sending messages from any node to all others. Any node can send the message, and they're all FTD. and i need the Mesh feature to work (hence why I am using OpenThread).

Reliable multicast is a difficult networking problem that is not specific to Thread. I realize now from your original message you are mostly concerned about multicast messaging. The primary challenge is dealing with the potential flood of ACKs coming back.

Link-local IPv6 multicast translates to a single IEEE 802.15.4 broadcast.

Jonathan, thanks for all the details.
If UDP multicast messages are translated to a single IEEE 802.15.4 broadcast, then I am not really taking advantage of the "mesh" features of OpenThread.
Do you have any recommendation on what's a better approach or layer to use? Or is it that I have to develop application on top of Thread to ensure message propagation throughout the network, regardless of what layer i am calling in my application? Currently i'm calling UDP API, but i can go up to CoAP if that helps?
is there no readily-available multicast protocol in OpenThread that I can use? Multicast seems such a generic use-case, I am surprised if it doesn't support multicast with propagation (retransmit via all FTDs) feature built in.

If UDP multicast messages are translated to a single IEEE 802.15.4 broadcast, then I am not really taking advantage of the "mesh" features of OpenThread.
Do you have any recommendation on what's a better approach or layer to use? Or is it that I have to develop application on top of Thread to ensure message propagation throughout the network, regardless of what layer i am calling in my application? Currently i'm calling UDP API, but i can go up to CoAP if that helps?
is there no readily-available multicast protocol in OpenThread that I can use? Multicast seems such a generic use-case, I am surprised if it doesn't support multicast with propagation (retransmit via all FTDs) feature built in.

As I tried to convey in https://github.com/openthread/openthread/issues/5347#issuecomment-670196678, realm-local and higher scope multicast in Thread uses MPL for forwarding over multiple hops. Thread's implementation of MPL works by having routers broadcast each newly-received message multiple times. It is a "flooding" protocol, but over a dynamically set of elected routers.
You also asked about reliable multicast delivery. MPL is best effort in that there are no acknowledgments. Reliable multicast delivery is a difficult problem because it typically involves bidirectional communication between the source and each group member.

I guess i don't fully understand the difference between Link-Local IPv6 and non-link-local.
There's no connectivity to the outside world out of my OpenThread network (e.g. no access from the internet is required) - 1 single PANid.
I am opening a UDP socket on all devices (every node is FTD), and i open a specific port number on all.
Then from the sender device I send a message to IPv6 address "FF03::1" which defines multicast to every node from what I understand.
Is that considered Local-Link or non-local-link?
Address is populated via : otIp6AddressFromString("FF03::1", &messageInfo.mPeerAddr);
Message is created using : otUdpNewMessage(thread_ot_instance_get(), NULL);
and finally sent to the socket via : otUdpSend(&socket, test_Message, &messageInfo);
all radio modes are THREAD_RADIO_MODE_RX_ON_WHEN_IDLE
auto-commissioning and auto-start are both on.

I guess i don't fully understand the difference between Link-Local IPv6 and non-link-local.

Thread devices link-local scope as devices that are reachable with a single radio transmission. If you want to communicate over multiple hops, then you need to use realm-local or larger scope. See Thread Primer - IPv6 Addressing for more information.

Sorry I was away for a week. Picking this back up again..
I read the primer - In that document there's no mention of "realm-local" or "larger scope".
I assume those refer to what the Thread Primer calls "Mesh Local" and its IPV6 address is "ff03::1" which I was using in my code.
Therefore I assume the Thread network was already using MPL retransmits that you mentioned. Set to 2 by default. I have roughly 80% success rate on multicast messages, nodes closer to the initiator of the multicast message had higher success rates.
I will increase that to 4 and see if my lossy network gets better success rate across the system. In theory it should.
Is there any way that I can observe or get metrics that show MPL retransmits are working? e.g. Via the CLI on each node?  I have intentionally reduced the RF output power to mimic a lossy network.

Unfortunately, the IPv6 MPL module does not currently expose any of its state via APIs or CLIs.
OpenThread does have some MAC-level counters that display the number of broadcast messages sent/received, but that will include any MAC frame that is broadcast, not just MPL message.
You could also enable logging to observe what messages are sent/received.
If you wanted to just extract the MPL information, you could look at adding log messages in src/core/net/ip6_mpl.cpp. Interesting areas to log would be:

When a new IPv6 multicast message is received (here)

When a buffered IPv6 multicast message is retransmitted (here and here)

Hope that helps.


Enhancement: Support NetVSC/VF dual NIC in hyper-V VMs

Motivation
On certain Hyper-V provisioned VMs, there're two NICs, one NetVSC, another VF. They can share same MAC address to act as single device from external point of view. Because VF is faster, it may be used to optimize data plane performance, while NetVSC is used for control plane. This is transparent to application layer if networking stack is done within the kernel. However, XDP bypasses the kernel, so we have to deal with it manually. This change would allow user to configure two interfaces for XDP to attach to and poll as incoming packets could come from either the VF or the NetVSC.

Looks fine to me. Is there a reason to keep the VF rx queues separate from the other rx queues? Could we just append them to the same vec?

I think we can merge them into one, although keeping them separate might serve some diagnostic purpose now or in the future.


Treat semicolons like commas in the gene ID upload tool

Currently this produces no results:
SPAC19G12.17; SPBP16F5.04;

I've fixed that.
I've also added a "Clear" button on the right because I've been meaning to for quite a while:


Failure to import tutorial data

For bugs, please provide the following:
Expected behavior
The pilosa import command give in the tutorial (https://www.pilosa.com/docs/getting-started/#sample-project) completes correctly.
Actual behavior
The import command fails to complete and gives an error message saying that there are too many files opened.  Attempting to re-execute the command gives the same error.  lsof | grep stargazer | wc -l reports 20,280 files opened by Pilosa.  Afterwards, Pilosa fails to start with the same error message.
$ ./pilosa import -i repository -f stargazer stargazer.csv

[ elided ]
2017/07/10 14:55:53 fragment: snapshotting repository/stargazer/inverse_20150823/0
2017/07/10 14:55:53 fragment: snapshot complete repository/stargazer/inverse_20150823/0 took 15.461138ms
2017/07/10 14:55:53 fragment: error flushing cache on close: err=open /home/vfoley/.pilosa/repository/stargazer/views/standard_20150825/fragments/0.cache: too many open files, path=/home/vfoley/.pilosa/repository/stargazer/views/standard_20150825/fragments/0
2017/07/10 14:55:53 import error: index=repository, frame=stargazer, slice=0, bits=50000, err=open cache: open /home/vfoley/.pilosa/repository/stargazer/views/standard_20150825/fragments/0.cache: too many open files

~$ lsof | grep stargazer | wc -l
20280

$ ./pilosa server
Pilosa v0.4.0, build time 2017-06-08T15:46:41+0000
Using data from: /home/vfoley/.pilosa
2017/07/10 15:04:03 opening index: repository
2017/07/10 15:04:03 fragment: error flushing cache on close: err=open /home/vfoley/.pilosa/repository/stargazer/views/inverse_20160314/fragments/0.cache: too many open files, path=/home/vfoley/.pilosa/repository/stargazer/views/inverse_20160314/fragments/0
Error: error running server: server.Open: opening Holder: open index: name=repository, err=open frame: name=stargazer, err=open view: view=inverse_20160314, err=open fragment: slice=%!s(uint64=0), err=open cache: open /home/vfoley/.pilosa/repository/stargazer/views/inverse_20160314/fragments/0.cache: too many open files
Usage:
  pilosa server [flags]

Flags:
      --anti-entropy.interval duration       Interval at which to run anti-entropy routine. (default 10m0s)
  -b, --bind string                          Default URI on which pilosa should listen. (default ":10101")
      --cluster.gossip-seed string           Host with which to seed the gossip membership.
      --cluster.hosts stringSlice            Comma separated list of hosts in cluster.
      --cluster.internal-hosts stringSlice   Comma separated list of hosts in cluster used for internal communication.
      --cluster.internal-port string         Port to which pilosa should bind for internal state sharing.
      --cluster.poll-interval duration       Polling interval for cluster. (default 1m0s)
      --cluster.replicas int                 Number of hosts each piece of data should be stored on. (default 1)
      --cluster.type string                  Determine how the cluster handles membership and state sharing. Choose from [static, http, gossip] (default "static")
  -d, --data-dir string                      Directory to store pilosa data files. (default "~/.pilosa")
      --log-path string                      Log path
      --max-writes-per-request int           Number of write commands per request. (default 5000)
      --metric.host string                   Default URI to send metrics.
      --metric.poll-interval duration        Polling interval metrics.
      --metric.service string                Default URI on which pilosa should listen. (default "nop")
      --plugins.path string                  Path to plugin directory.
      --profile.cpu string                   Where to store CPU profile.
      --profile.cpu-time duration            CPU profile duration. (default 30s)

Global Flags:
  -c, --config string   Configuration file to read from.

error running server: server.Open: opening Holder: open index: name=repository, err=open frame: name=stargazer, err=open view: view=inverse_20160314, err=open fragment: slice=%!s(uint64=0), err=open cache: open /home/vfoley/.pilosa/repository/stargazer/views/inverse_20160314/fragments/0.cache: too many open files

Steps to reproduce the behavior
Install Pilosa v0.4.0 and follow the tutorial.
Information about your environment (OS/architecture, CPU, RAM, cluster/solo, configuration, etc.)

Lenovo Thinkpad T450 laptop
OS: Debian GNU/Linux 9 (Stretch)
Kernel: 4.9.0-3-amd64
CPU: Intel(R) Core(TM) i5-5300U CPU @ 2.30GHz
RAM: 12 GiB

Thanks for the report @gnuvince - we've got a note in the docs about setting open file limits, and if you run ulimit -n 262144 in the terminal right before you run Pilosa, that will probably solve your immediate issue, but this keeps coming up, and I think I have a better solution.
We can use Go's syscall package like so:
    lim := &syscall.Rlimit{}
    syscall.Getrlimit(syscall.RLIMIT_NOFILE, lim)
    fmt.Printf("lim: %v\n", lim)
    err := syscall.Setrlimit(syscall.RLIMIT_NOFILE, &syscall.Rlimit{Cur: 10000, Max: 10000})
    fmt.Println("sycall.Setrlimit: ", err)
    lim = &syscall.Rlimit{}
    syscall.Getrlimit(syscall.RLIMIT_NOFILE, lim)
    fmt.Printf("lim: %v\n", lim)

to set open file limits from within Pilosa. This will have to be tested on multiple platforms since syscall is OS specific, although the above works on my mac. If we can't set the limit very high, we should log a warning so that the user knows she'll need to get privileges to set the limit higher or whatever.

I created https://github.com/pilosa/pilosa/issues/722 for implementation of the long term fix. @gnuvince if the ulimit fix I described works for you, please go ahead and close this ticket - if not, let us know and we'll dig in further.

Yep, the ulimit workaround fixes the immediate issue, thanks!

Use ulimit -a to list the current configurations. And under some versions of OS X ulimit -n 262144 just produces error: cannot modify limit: Invalid argument. We may use ulimit -n<PHONE_NUMBER>44 instead.


Run e2e training test without 4 bit quant

Set training options to use GMU RAM to fit in a Tesla T4.

@russellb I'm not sure how to trigger the e2e test, or if I can. would you mind kicking it off please

@russellb I'm not sure how to trigger the e2e test, or if I can. would you mind kicking it off please

The instructions are here, but you need write access to the repo to do it.  https://github.com/instructlab/instructlab/blob/main/docs/ci.md
I'll kick it off now

The e2e run passed here: https://github.com/instructlab/instructlab/actions/runs/8947293535
However, it also tripled the runtime (and cost) of the job. ~30 min to ~1h30min -- or ~$2 to ~$6 USD per run.
We should think about what we can do to bring the time back down.

The e2e run passed here: https://github.com/instructlab/instructlab/actions/runs/8947293535
However, it also tripled the runtime (and cost) of the job. ~30 min to ~1h30min -- or ~$2 to ~$6 USD per run.
We should think about what we can do to bring the time back down.

Ack, agreed, not sure the additional runtime would be justified, I wonder if we can add timestamps to the job logs so we can figure out which part of the training took all the time. Not sure I can make it shorter but should be able to figure out where we are taking the time.

The e2e run passed here: https://github.com/instructlab/instructlab/actions/runs/8947293535
However, it also tripled the runtime (and cost) of the job. ~30 min to ~1h30min -- or ~$2 to ~$6 USD per run.
We should think about what we can do to bring the time back down.

Ack, agreed, not sure the additional runtime would be justified, I wonder if we can add timestamps to the job logs so we can figure out which part of the training took all the time. Not sure I can make it shorter but should be able to figure out where we are taking the time.

I just put this up to add some more timestamps -- https://github.com/instructlab/instructlab/pull/1139

The e2e run passed here: https://github.com/instructlab/instructlab/actions/runs/8947293535
However, it also tripled the runtime (and cost) of the job. ~30 min to ~1h30min -- or ~$2 to ~$6 USD per run.
We should think about what we can do to bring the time back down.

Ack, agreed, not sure the additional runtime would be justified, I wonder if we can add timestamps to the job logs so we can figure out which part of the training took all the time. Not sure I can make it shorter but should be able to figure out where we are taking the time.

I just put this up to add some more timestamps -- #1139

thanks, lgtm for a quick ref but I was thinking of line by line timestamps, I now see that line by line would be redundant as github provides it (in the results output page, click settings icon in the top right and then show timestamps)

The e2e run passed here: https://github.com/instructlab/instructlab/actions/runs/8947293535
However, it also tripled the runtime (and cost) of the job. ~30 min to ~1h30min -- or ~$2 to ~$6 USD per run.
We should think about what we can do to bring the time back down.

looks like its spending most of its time in testing the train model, this is slow and we are doing 10 of them, I've reduced the number for 1 for training to see what happens.
Also I see that my trainging args PR needed a rebase, I've done this for testing but I need to fix it properly (I'll do that later)

@russellb would you mind triggering the job on this again please

@russellb would you mind triggering the job on this again please

https://github.com/instructlab/instructlab/actions/runs/9063176914

@russellb would you mind triggering the job on this again please

https://github.com/instructlab/instructlab/actions/runs/9063176914

Sorry, I stomped over the sh->bash change, I've fixed it now

@russellb would you mind triggering the job on this again please

https://github.com/instructlab/instructlab/actions/runs/9063176914

Sorry, I stomped over the sh->bash change, I've fixed it now

No worries. I really need to come up with an easier way to allow a broader group of people to run this on demand.
https://github.com/instructlab/instructlab/actions/runs/9071407952

No worries. I really need to come up with an easier way to allow a broader group of people to run this on demand.
https://github.com/instructlab/instructlab/actions/runs/9071407952

thanks, once more (sorry for being a pain)

@Mergifyio rebase

No worries. I really need to come up with an easier way to allow a broader group of people to run this on demand.
https://github.com/instructlab/instructlab/actions/runs/9071407952

thanks, once more (sorry for being a pain)

not a pain!
https://github.com/instructlab/instructlab/actions/runs/9146410728

38 minutes, still a little longer then the other run but not longer an hour longer

38 minutes, still a little longer then the other run but not longer an hour longer

Nice, that's an acceptable increase to me for the added coverage.

Closing this, a lot has changed since it was created and its probably no longer relevant


Switch transpiler to babel and rollup

:exclamation: Not ready for merging :exclamation:
This is the start of replacing esperanto with babel and rollup. This transpiles the entire src directory into build/cjs in a commonjs module format, and uses that to run unit tests. It also builds some umd artifacts currently, though I am missing quite a few still.
TODO:

[ ] Create locales.js
[ ] Create moment-locales.js
[ ] Custom locales builder.
[ ] Unit tests for browser
[ ] Retain comment headers in built files

A potentially breaking change is that the locale files no longer export default. I'm not sure anyone was using them as var es = require('moment/locale/es') anywhere. I changed this so that rollup wouldn't complain about not declaring a module name for each of those files, I'm not sure if there is another way around this.

The src/locales.js and src/test/all.js -- can we autogenerate those when we need to use them, I hope they won't be part of the repo when this is merged.

See also #2980

@timrwood do you want help on this? Can't say it's my area of expertise, but I'd like to see it done. Tell me what to do and I'll play with it a bit.

Closing. Iskren finished in different branch.


Command Group Implementation

Grouping the Command objects will enable the application to run multiple tasks at the same time.
This will showcase the potential of this development very efficiently

The Test Case test_CommandGroupProfiling documents very effectively that enabling Command.read_timeout renders useless any profiling at the CommandGroup level:
Command (1641) 'command-script:3s':
Read Timeout: '2'
Execution Time: '6.0432939529418945'
ERROR CODE: '0'
EXIT CODE: '0'
STDOUT:
'Start - Time Now: '1619689885.3469896'
Number of arguments: 2 arguments.
Argument List: ['/home/runner/work/pycommand/pycommand/tests/command_script.py', '3']
test script absolute path: '/home/runner/work/pycommand/pycommand/tests/command_script.py'
script 'command_script.py' START 0
script 'command_script.py' PAUSE '3' ...
script 'command_script.py' END 1
End - Time Now: '1619689888.3473907'
script 'command_script.py' done in '3000.401020050049' ms
script 'command_script.py' EXIT '0'
'
STDERR:
'script 'command_script.py' START 0 ERROR
script 'command_script.py' END 1 ERROR
'

The test script command_script.py documents that it took only 3 s to complete the job but the child process was not reaped untill 6 s of overall test execution time.
This happens because the Manager Process did not check it untill it has not completed the whole check cycle.
Only with the setting of Command.read_timeout == 0 and having the Check Interval disabled in the CommandGroup you are able to measure the Child Process Execution more reliably.
The "Check Interval" is a setting that is meant to lighten the host load for jobs that naturally take longer to complete and where real-time execution is not required.


rplidar_ros: 1.5.2-0 in 'jade/distribution.yaml' [bloom]

Increasing version of package(s) in repository rplidar_ros to 1.5.2-0:

upstream repository: https://github.com/robopeak/rplidar_ros.git
release repository: https://github.com/robopeak/rplidar_ros-release.git
distro file: jade/distribution.yaml
bloom version: 0.5.21
previous version for package: null

rplidar_ros
1.5.1 (2016-04-12)
* Release 1.5.1.
* Add the deiver support rplidar A2.
* Contributors: yhZhao, RoboPeak Public Repos

Releasing new package due to low impact on buildfarm.

The source deb is broken at least on Utopic: http://build.ros.org/job/Jsrc_uU__rplidar_ros__ubuntu_utopic__source/10/console
I'm going to revert until that can be sorted out. It looks like the control file is malformed.

For the record: the sourcedeb job failed for all platforms.

@kintzhao I'm not sure why your source packages generated by bloom are not correct. Looking at the release repository there is a strange thing that has happened where the previous versions released (in order) are 1.5.2-0, 1.5.2-1 and then 1.5.2-0 again. This is really strange.
You should only need to run bloom-release -r jade -t jade rplidar_ros, answer the questions, and let it open a pull request. It seems like you did something like this, but it also seems something strange happened as well. Please run it again and it should open a pr here for 1.5.2-2.

@wjwwood  I  run  bloom-release -r jade -t jade rplidar_ros   , but the version is 1.5.2-1 .


XGBoostError: src/predictor/predictor.cc:55: Unknown predictor type gpu_predictor'

For bugs or installation issues, please provide the following information.
The more information you provide, the more easily we will be able to offer
help and advice.
Environment info
Operating System: Windows 10 64Bits
Compiler:
Package used (python/R/jvm/C++): Python
xgboost version used: 0.16
If installing from source, please provide

The commit hash (git rev-parse HEAD): ff9180cd73f21c21b8852b010dd664966bb00970
Logs will be helpful (If logs are large, please upload as attachment).

If you are using jvm package, please

add [jvm-packages] in the title to make it quickly be identified
the gcc version and distribution

If you are using python package, please provide

The python version and distribution: 3.5 Anaconda
The command to install xgboost if you are not installing from source

If you are using R package, please provide

The R sessionInfo()
The command to install xgboost if you are not installing from source

Steps to reproduce
space ={
'booster '    : 'gbtree',
'objective'   : 'binary:logistic',
'eval_metric' : 'auc',
'max_depth'   : hp.choice("max_depth",        np.arange(4, 7,    dtype=int)),
'tree_method' : 'gpu_exact',
'alpha'            : hp.uniform('alpha', 1e-6, 1e-4 ),
'lambda'           : hp.uniform('lambda', 1e-6, 1e-4 ),
'subsample'        : hp.uniform('subsample',0.6, 0.9),
'min_child_weight ': hp.uniform('min_child_weight', 0.6, 0.8),
'learning_rate'    : hp.uniform('learning_rate', 1e-3, 1e-2),
'seed'            : hp.randint('seed',2000000)

}
xgb_params = sample(space)
gbm_model = xgb.train(
xgb_params,
xgb_train,
num_boost_round=10,
# metrics='mlogloss',
# valid_names=None,
# fobj=None,
# init_model=None,
# feature_name='auto',
# categorical_feature='auto',
# early_stopping_rounds=None,
# evals_result=None,
verbose_eval=False
# learning_rates=None,
# keep_training_booster=False,
# callbacks=None)
)
What have you tried?

Reinstall from source
Change gpu_exact to gpu_hist
Google it

It seems that though cmake setting "PLUGIN_UPDATER_GPU" is deprecated it is still used in
make/config.mk to determine if plugin/updater_gpu/plugin.mk should be added.
As a workaround try
$ cmake .. -DPLUGIN_UPDATER_GPU=ON

I'm having the same issue with Fedora 26. I cloned xgboost from git  and I got the same problem. I did run CMAKE with -DPLUGIN_UPDATER_GPU=ON. Everything compiled ok.
When I run the benchmark.py in the tests folder, I got this message:
Training with 'gpu_hist'
[11:33:29] /root/xgboost/dmlc-core/include/dmlc/./logging.h:300:
[11:33:29] /root/xgboost/src/predictor/predictor.cc:55: Unknown predictor type gpu_predictor
Best regards,
Lorenzo

The error 'Unknown predictor type gpu_predictor' indicates that XGBoost has not been installed with GPU support. If you are able to successfully compile with GPU support please check that this was installed correctly and that there are no other existing versions of XGBoost on your system.


Global definition of sources

It would be nice if there was a way to set these globally through the modal config or the like.
https://github.com/sahusoftcom/plupload-angular-directive/blob/master/src/js/plupload-angular-directive.js#L40-L43

Hi,
I am trying to set the upload URL through controller of the page, as per the requirement the upload url will be dynamic. I assigned the url value to the $scope like
$scope.uploadUrl=someVariable
and then tried to use it like
pl-url="uploadURL"
but its not able to assign the dynamic url value somehow can you please suggest some way by which I can do it.

Hi,
pl-url attribute takes string as an input. So, you must use
pl-url="{{uploadUrl}}" to use the url from a scope variable.
Thanks.
On Fri, Sep 11, 2015 at 9:17 PM, ektatechila<EMAIL_ADDRESS>wrote:

Hi,
I am trying to set the upload URL through controller of the page, as per
the requirement the upload url will be dynamic. I assigned the url value to
the $scope like
$scope.uploadUrl=someVariable
and then tried to use it like
pl-url="uploadURL"
but its not able to assign the dynamic url value somehow can you please
suggest some way by which I can do it.
—
Reply to this email directly or view it on GitHub
https://github.com/sahusoftcom/plupload-angular-directive/issues/1#issuecomment-139581638
.


[automation] Introduce flexible deploy loop

This new tasks file will allow a more flexible approach for VA
deployment.
It will rely on the automation file provided in the architecture
repository, and execute each stage.
It also allows to stop the loop in 3 "strategic" steps, for each
described stage.
Depends-On: https://github.com/openstack-k8s-operators/ci-framework/pull/1167
As a pull request owner and reviewers, I checked that:

[ ] Appropriate testing is done and actually running
[X] Appropriate documentation exists and/or is up-to-date:

[X] README in the role

/test all

recheck

/test all

/approve

/hold
/lgtm

/hold cancel


found 1 high severity vulnerability (trim-newlines)

High            Regular Expression Denial of Service
Package         trim-newlines
Patched >=3.0.1 <4.0.0 || >=4.0.1
Dependency of   @electron-forge/cli [dev]
Path            @electron-forge/cli > @electron-forge/core > nugget >
pretty-bytes > meow > trim-newlines
am running on 6.0.0-beta.57
Running npm audit fix would not upgrade

This vulnerability 100% does not affect Electron Forge. TL;DR: ignore this specific vulnerability with this dependency tree.
Details
nugget is used by electron-forge install to install Electron apps for the host platform/arch. It depends on pretty-bytes as a library for showing numan-readable sizes of downloaded files. pretty-bytes depends on meow, which is a CLI helper, and is used only for its CLI, which nugget does not use. meow, in turn, depends on trim-newlines (the reason for this is not interesting in this case).
To reiterate, the code in this dependency tree for trim-newlines will never get called because it's for a CLI that's never run.
See also
npm audit: Broken by Design


Format

This should cover it. 😑

I know we're moving fast, but for the record, I think a PR touching so many files in so many ways should have had more reviewers and more discussion... /grumpy-old-man


Make logging scalable

e.g. rotate log files hourly

Looking...

@swapagarwal Can you please provide your review of the provided PR? Thanks