Support proper zooming

Currently, terminal text does not scale with browser zoom level.
When editing a terminal, Ctrl+scroll wheel changes the browser zoom level.

You can zoom in sshx with the scroll wheel, or pinch and drag gestures on trackpad / mobile.


Version 1.0.11 keeps forgetting rooms

weird issue has cropped up with latest version where all room allocations are forgotten after a couple of days.

Hi,
Can you remember me what controller are you using?
Cause under all tests with Home I had no issues but another user noticed this with Alexa upgrading to matterbridge 1.6.1 from 1.6.0.
Can you notice if the endpoint numbers got changed?
Matter.js updated some code there that can explain this.

Using a raspberry pi.
HomeKit 18.2
It's happened again tonight
Can I send logs?

Sure send it but I don’t understand iit happens randomly or you did something?
The update of today for instance has been carefully tested and even on my production setup had no issues at all. And I have more then 100 devices there.

It just seems to happen a day or so afterwards.
I'm away for a few weeks but will send some logs when back

Feel free to post then when you are back.


Fix error message for key not matching allowed pattern

Fixes #12
This PR makes the error message more accurately reflect what is checked.
Also fixes grammar error.

/cc @pohly


Frame Application Screen: Order Form

User story

As a facility manager
I want / need a screen with an order form that I can submit
So that I can create repairing/maintenance orders for an asset of a certain facility

Acceptance criteria

[ ] The screen needs to be connected to the correct route in the navigation bar
[ ] Upon visiting the page/screen, the facility manager sees a form
[ ] The form needs to have the following fields, two select fields, one containing a dropdown of all the facilities, and the second one containing all the assets of the selected facility (the user's first facility is set as default), a phone field, an email field, and a textfield for additional messages. Every field except the textfield is required.
[ ] The form must be submittable already, and its values must be displayed (f.e. in the console)
[ ] The screen must use components from Siemens iX

DoD general criteria

[ ] Feature has been fully implemented
[ ] Feature has been merged into the mainline
[ ] All acceptance criteria were met
[ ] Product owner approved features
[ ] All tests are passing
[ ] Developers agreed to release

Dependencies

Item #48
Item #47

@shahraz1998  Patrick and I are not sure about the second dropdown form containing the assets. Right now in the testdata we created there are no assets. Since @Hydraneut and @HaruspexSan are creating more test data in this issue https://github.com/orgs/amosproj/projects/43/views/2?pane=issue&itemId=62977084 it would be good to first have defined what data a facility needs to have.
We think you are referring to these assets described in the Siemens data but we are not sure how the facilities are related to them. Is it correct that a facility must have at least one asset and can as many as it wants. And are assets like described here https://miro.com/app/board/uXjVKOVn0cI=/?moveToWidget=3458764588064757504&cot=14

Definition of work orders
@shahraz1998 apart from the thing with assets I have more questions regarding work orders maybe this probably also needs to be discussed with @sn0wcat.
Right now we have just created them like this
export interface IOrderMock {
    id: string;
    facilityId: number;
    icon: string;
    notification: string; // text next to the icon
    heading: string;
    subheading: string;
    variant: string; // success(green)/warning(yellow)/critical(orange)
}

You can see a lot of attributes were picked to fill the push card with content. But should we even use push cards to display these? Or is there something better on the Siemens iX website

In any case we need to make sure we collect all that data in the order form or remove attributes we don't ask for there. For example the 'heading', 'subheading' and 'notificaton' field were mostly to fill this card with content but probably don't make sense to keep.
In the Acceptance criteria Shahraz you also mentioned the following fields so we will add them to the work order data:
phone number, email address, textfield for additional messages
We need to know if a work order has more data so f.e. a status (completed/ progressing/...could maybe be shown using a chip)
and also the corresponding asset for the work order (in case one work order is for one asset)
Do we need a detailed view for each work order where you can see f.e. it status? Or is it enough if we just see these things on the same site without changing the url after clicking on them? Maybe using the Flip Component from the ix website or some other form of dropdown view?

Most questions were resolved in the IP meeting today. I probably overworried a little


Issue 246 internationalize date picker

Internationalize DatePicker.

Translated strings
Support date format in other locales

Resolves #246
Thanks for contributing to Terra.
@cerner/terra

The Packages Requiring I18n in the project ReadMe needs updated as well.

The  Packages Requiring I18n section in the ReadMe already included DatePicker. But your commented reminded me that I need to add Internationalization Support in the Component Features section in the component ReadMe.

+1 provided the merge conflicts are resolved.

Functional verification completed. All date picker examples display and function correctly in each supported locale.


Show a new messages divider in secure conversations chat transcript

Jira issue:
https://glia.atlassian.net/browse/MOB-1735
Additional info:
When fetching chat transcript, there is a field called delivered_at which can have a date, or can be nil. If it’s nil, it means that the message hasn’t been read yet. Above the oldest unread message, there should be a new message divider that looks like the document in Figma. This component has to be fully translatable and accessible.
Screenshots:
https://user-images.githubusercontent.com/19994238/220933523-63988f41-a6b9-4eea-9af9-1c5b49be4f7b.mp4

Good job 👍

!squash

!merge


Add --json flag for json output/input

It's easier to use namerctl programmatically if the format is json. This adds the --json flag for dtab commands.

My preference is to review & merge this with failing tests, since the act of merging to master will resolve the issue

@olix0r sure, we can move to always talking to the json api everywhere. It irks me to have to parse the json response only to re-marshal it for the CLI, which is why I avoided it.


Incompatible with 0.8.5

Causes a black screen on every scene, even if no enhanced terrain was drawn.

Working on it.
0.8.x changed some fundamental things that's going to take some time to fix.

This should be fixed with the latest version.
I'm just slow to close tickets.  :-)


feat: confirm overwrite when importing

SUMMARY

This PR changes the import API to require an explicit flag in order to overwrite existing assets.
BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

N/A
TEST PLAN

Added unit tests, both for the commands and the API.
ADDITIONAL INFORMATION

[ ] Has associated issue:
[ ] Changes UI
[ ] Requires DB Migration.
[ ] Confirm DB Migration upgrade and downgrade tested.
[ ] Introduces new feature or API
[ ] Removes existing feature or API

Codecov Report

Merging #9 (a88876d) into consolidate_import_logic (2297f9b) will decrease coverage by 4.72%.
The diff coverage is n/a.

@@                     Coverage Diff                      @@
##           consolidate_import_logic       #9      +/-   ##
============================================================
- Coverage                     67.67%   62.94%   -4.73%
============================================================
  Files                           939      473     -466
  Lines                         45408    16920   -28488
  Branches                       4361     4361
============================================================
- Hits                          30730    10651   -20079
+ Misses                        14575     6092    -8483
- Partials                        103      177      +74

Flag
Coverage Δ

cypress
?

javascript
62.94% <ø> (ø)

python
?

Flags with carried forward coverage won't be shown. Click here to find out more.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2297f9b...a88876d. Read the comment docs.


Docs: explain how to use callbacks

Hi,
a huge part of Cursive I don't understand is Callbacks. What are they for? What is the advantage of returning a callback in response to consuming event?
Can you please explain it a little better in docs?

Returning a callback is a bit strange but it's how you get access to the Cursive object while keeping the initial callback static.

Here's a brief summary of the idea behind the design behind View::on_event and EventResult:

View::on_event takes a &mut self so it can modify the view reacting to the event.
As reaction to the event, we want to potentially update the view tree; this would need a &mut Cursive
But the root Cursive contains the view, so from &mut Cursive I could get another &mut my_view. This is impossible by rust standard. For example, what if I remove the view itself in the callback? The view cannot really be dropped since we're still in one of its methods.

The conclusion is that we cannot easily[1] take a &mut Cursive in View::on_event.
Instead, we can prepare "what we would do if we had this &mut Cursive" - that's the EventResult. You split your event response in two parts: the first half (the body of View::on_event) that updates the current view, and the second part (the EventResult) that updates the Cursive root to modify the view tree.
Most of the times it's easy to split (one of the two halves may be simply empty). In more complex cases where you want to update the current view based on something in the view tree, you'll just need to somehow update the current view from the EventResult itself. This can be done by sharing a Rc with the callback, or by re-finding the current view from the root if it has an ID or something.
In the future I want to add a "ViewPath" to the on_event method, which describes the path in the view tree at the moment the event is dispatched. It would then be easy to give this path to the EventResult to find the current view again, without the need for a NamedView.
Another long-term goal I had in mind was EventResult not meant for the Cursive root, but for one of the parent views. This would potentially allow "sub-root" views to sandbox children, or enable more inter-views communications. This is still a vague idea and will obviously need a lot more design to get anywhere near usable.
I'll update a bit the documentation for View::on_event to make some of this clearer.
[1]: One convoluted way to have both would be to wrap every single view in some sort of Rc + RefCell; though it would need a deep refactor of how the view tree is visited for events. Also it might add some (probably small) overhead to every single view, including wrapper views, which is something I'm trying to avoid (wrapper views are a good idea as long as they are really cheap).


graph issue

When zooming out, the timeline does not become appropriately sized, and it is not movable to see more history. Half the graph space can turn empty.

Works for me.
Click and drag the graph.

You're right. I guess I expected that if I scrolled sideways it should do the same. Didn't realize must click and drag. Thanks.


Double execution of integration test

Expected Behavior
Test is executed once

Current Behavior
Test is executed twice

Failure Information (for bugs)
TestUpdateConnectionOnNSEChange:
https://circleci.com/gh/networkservicemesh/networkservicemesh/88280#tests/containers/0

Also seems fixed by @artem-belov


Continuous access evaluation resulted in challenge with result: InteractionRequired and code: TokenIssuedBeforeRevocationTimestamp

This is autogenerated. Please review and update as needed.
Describe the bug
When calling any 'az ad' request, I receive "Continuous access evaluation resulted in challenge with result: InteractionRequired and code: TokenIssuedBeforeRevocationTimestamp".
It it not obvious to me what I can do to fix this... Recently, I have updated my company's AD password, which lauched a new login in the azure portal. I completed it with MFA successfully, and I can access all information in the portal, but when I use az cli, I get this error. I have already done az login to make sure I had an updated token.
What am I supposed to do to fix this?
Command Name
az ad signed-in-user show
Errors:
"Continuous access evaluation resulted in challenge with result: InteractionRequired and code: TokenIssuedBeforeRevocationTimestamp"
To Reproduce:
Steps to reproduce the behavior. Note that argument values have been redacted, as they may contain sensitive information.

Put any pre-requisite steps here...
az ad signed-in-user show

Expected Behavior
successful request
Environment Summary
Linux-5.15.0-71-generic-x86_64-with-glibc2.31, Ubuntu 20.04.5 LTS
Python 3.10.10
Installer: DEB

azure-cli 2.48.1

Additional Context

Thank you for opening this issue, we will look into it.

Have you tried to re-login by running az login?

Hi! I have tried az login, which prompts login, but I have not tried with az logout in between. I will try that

Yeah, just happened here and
az logout
az login

Seems to have fixed it or it was a hickup.

Confirming, the problem persists with doing only az login and disappered after doing both az logout and az login.


Add ability to validate TLS certificate chain for Deploy-Service API calls

Currently, HTTPS/TLS validation for API calls to deploy-service is turned off via a hardcoded constant. This PR adds the config ability to enable TLS certificate verification within the global config to prevent MITM attacks etc.
Default config behavior retains the existing policy of not validating the certificate chain.

LGTM @yongwen


Distribution includes spread operator and is not compatible with IE

Sorry for the short description--I'm assuming that the title is enough :)

@jurca Hello. Any chance you could take a look at this and the pull request? Thanks!

Hello and sorry for taking so long to reply.
Since this does not increase the size of the generated code much, I'm going to accept this change. Thanks for the PR!

I've just published<EMAIL_ADDRESS>with your PR applied.
Thank you for your contribution!

Thank you for the quick turnaround. It's much appreciated!


fix: use arch appropriate buffer sizes

Another one for https://github.com/ssbc/go-muxrpc/issues/13.
Follows approach of https://github.com/dgraph-io/ristretto/pull/238 - we limit the buffer size to something still big but in range for 32 bit archs. I don't have a 32 bit machine to test this range but the PR above shows another large / in-production project using buffer size, so it seems legit :woman_shrugging:

@staltz think this is good to go now :mushroom:


Different Sound coming if playing in TaalMala and web demo

Different Sound coming if playing in TaalMala and web demo
We are trying to play this -
Dhin [.5] | Ti [0.125] | Ra | Ki | Ti | Dhin [.5] | Na | Tun | Na | Kat+ | Na | Ti [0.125] | Ra | Ki | Ti | Dhi [.5] | Na | Dhi | Dhi | Na
Its sounding different

Where the scale is defined like its playing inn C# or what?


[DCOS-57125] Add --squash to docker build command to reduce CI runtime

See DCOS-57125 for more details

@vespian PTAL


Upgraded to Spring-Cloud-Config 2.1.0 and Spring-Boot 2.0.

Updated, let me know if that works for you.

Updated my commit email address as well...


DownloadManager doesn't rewrite files

Android's DownloadManger doesn't rewrite files. However, re-downloading course files is an option in the app. When a file is re-downloaded, the filename is appended with a number. Viewing the file from within the app will result in the first (not latest) download to be opened. This is not the behaviour a user would expected. The app should rewrite previous downloads (or rename the previous download).

After further investigation, it seems it is not possible for a teacher to re-upload a file on Moodle. Or delete a single file out of many uploaded. The only possibility seems to be deleting the entire module/forum discussion. Setting this as a low priority bug for now. Nevertheless, it is important that this is fixed, in case future version of Moodle allows deletion/reupload of files.

Re-labeled medium priority. It is important for re-downloads to overwrite files/file viewing to open latest file since it is a possibility that the download becomes corrupted.


Get stuck at "Added broadcast_0_piece0 in memory on" while runing Spark standalone cluster

Environment:

Python version 3.6.9
Spark version 3.0.0
TensorFlow version 2.6.2
TensorFlowOnSpark version 2.2.4
Cluster version Standalone, Hadoop 3.1.3

Describe the bug:
Get stuck at "INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on" while runing Spark standalone cluster while Training MNIST using Keras.
Logs:
2021-12-27 10:51:01,579 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2021-12-27 10:51:03,012 INFO spark.SparkContext: Running Spark version 3.0.0
2021-12-27 10:51:03,044 INFO resource.ResourceUtils: ==============================================================
2021-12-27 10:51:03,045 INFO resource.ResourceUtils: Resources for spark.driver:
2021-12-27 10:51:03,045 INFO resource.ResourceUtils: ==============================================================
2021-12-27 10:51:03,045 INFO spark.SparkContext: Submitted application: mnist_keras
2021-12-27 10:51:03,081 INFO spark.SecurityManager: Changing view acls to: amax
2021-12-27 10:51:03,081 INFO spark.SecurityManager: Changing modify acls to: amax
2021-12-27 10:51:03,081 INFO spark.SecurityManager: Changing view acls groups to:
2021-12-27 10:51:03,081 INFO spark.SecurityManager: Changing modify acls groups to:
2021-12-27 10:51:03,081 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(amax); groups with view permissions: Set(); users  with modify permissions: Set(amax); groups with modify permissions: Set()
2021-12-27 10:51:03,232 INFO util.Utils: Successfully started service 'sparkDriver' on port 45175.
2021-12-27 10:51:03,255 INFO spark.SparkEnv: Registering MapOutputTracker
2021-12-27 10:51:03,275 INFO spark.SparkEnv: Registering BlockManagerMaster
2021-12-27 10:51:03,289 INFO storage.BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
2021-12-27 10:51:03,289 INFO storage.BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
2021-12-27 10:51:03,291 INFO spark.SparkEnv: Registering BlockManagerMasterHeartbeat
2021-12-27 10:51:03,298 INFO storage.DiskBlockManager: Created local directory at /tmp/blockmgr-1a9cd977-b478-4add-bb24-887f1cb3e057
2021-12-27 10:51:03,311 INFO memory.MemoryStore: MemoryStore started with capacity 14.2 GiB
2021-12-27 10:51:03,320 INFO spark.SparkEnv: Registering OutputCommitCoordinator
2021-12-27 10:51:03,401 INFO util.log: Logging initialized @2665ms to org.sparkproject.jetty.util.log.Slf4jLog
2021-12-27 10:51:03,443 INFO server.Server: jetty-9.4.z-SNAPSHOT; built: 2019-04-29T20:42:08.989Z; git: e1bc35120a6617ee3df052294e433f3a25ce7097; jvm 1.8.0_212-b10
2021-12-27 10:51:03,457 INFO server.Server: Started @2722ms
2021-12-27 10:51:03,474 INFO server.AbstractConnector: Started ServerConnector@1627cd3b{HTTP/1.1,[http/1.1]}{<IP_ADDRESS>:4040}
2021-12-27 10:51:03,475 INFO util.Utils: Successfully started service 'SparkUI' on port 4040.
2021-12-27 10:51:03,495 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@5064c620{/jobs,null,AVAILABLE,@Spark}
2021-12-27 10:51:03,496 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@4a32f47a{/jobs/json,null,AVAILABLE,@Spark}
2021-12-27 10:51:03,497 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@3508d47e{/jobs/job,null,AVAILABLE,@Spark}
2021-12-27 10:51:03,498 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@7264cdd7{/jobs/job/json,null,AVAILABLE,@Spark}
2021-12-27 10:51:03,499 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@4a3d89f5{/stages,null,AVAILABLE,@Spark}
2021-12-27 10:51:03,499 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@7da95106{/stages/json,null,AVAILABLE,@Spark}
2021-12-27 10:51:03,500 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@46687b37{/stages/stage,null,AVAILABLE,@Spark}
2021-12-27 10:51:03,501 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@656b0411{/stages/stage/json,null,AVAILABLE,@Spark}
2021-12-27 10:51:03,501 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@60ad7cfe{/stages/pool,null,AVAILABLE,@Spark}
2021-12-27 10:51:03,502 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@46482ae5{/stages/pool/json,null,AVAILABLE,@Spark}
2021-12-27 10:51:03,502 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@1ef317fc{/storage,null,AVAILABLE,@Spark}
2021-12-27 10:51:03,502 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@15492f0a{/storage/json,null,AVAILABLE,@Spark}
2021-12-27 10:51:03,503 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@10ab3ccf{/storage/rdd,null,AVAILABLE,@Spark}
2021-12-27 10:51:03,503 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@57093dec{/storage/rdd/json,null,AVAILABLE,@Spark}
2021-12-27 10:51:03,504 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@4debbe0{/environment,null,AVAILABLE,@Spark}
2021-12-27 10:51:03,504 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@327dac75{/environment/json,null,AVAILABLE,@Spark}
2021-12-27 10:51:03,504 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@5b91ba46{/executors,null,AVAILABLE,@Spark}
2021-12-27 10:51:03,505 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@c385c3{/executors/json,null,AVAILABLE,@Spark}
2021-12-27 10:51:03,505 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@74bf8e9e{/executors/threadDump,null,AVAILABLE,@Spark}
2021-12-27 10:51:03,506 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@232553eb{/executors/threadDump/json,null,AVAILABLE,@Spark}
2021-12-27 10:51:03,510 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@5b24a17f{/static,null,AVAILABLE,@Spark}
2021-12-27 10:51:03,510 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@1c7efc23{/,null,AVAILABLE,@Spark}
2021-12-27 10:51:03,511 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@eeaa0a8{/api,null,AVAILABLE,@Spark}
2021-12-27 10:51:03,511 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@1823b1f1{/jobs/job/kill,null,AVAILABLE,@Spark}
2021-12-27 10:51:03,512 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@6c274337{/stages/stage/kill,null,AVAILABLE,@Spark}
2021-12-27 10:51:03,513 INFO ui.SparkUI: Bound SparkUI to <IP_ADDRESS>, and started at http://amax11:4040
2021-12-27 10:51:03,654 INFO client.StandaloneAppClient$ClientEndpoint: Connecting to master spark://<IP_ADDRESS>:7077...
2021-12-27 10:51:03,711 INFO client.TransportClientFactory: Successfully created connection to /<IP_ADDRESS>:7077 after 36 ms (0 ms spent in bootstraps)
2021-12-27 10:51:03,793 INFO cluster.StandaloneSchedulerBackend: Connected to Spark cluster with app ID app-20211227105103-0014
2021-12-27 10:51:03,794 INFO client.StandaloneAppClient$ClientEndpoint: Executor added: app-20211227105103-0014/0 on worker-20211226140545-<IP_ADDRESS>-43273 (<IP_ADDRESS>:43273) with 1 core(s)
2021-12-27 10:51:03,796 INFO cluster.StandaloneSchedulerBackend: Granted executor ID app-20211227105103-0014/0 on hostPort <IP_ADDRESS>:43273 with 1 core(s), 27.0 GiB RAM
2021-12-27 10:51:03,796 INFO client.StandaloneAppClient$ClientEndpoint: Executor added: app-20211227105103-0014/1 on worker-20211226140545-<IP_ADDRESS>-36031 (<IP_ADDRESS>:36031) with 1 core(s)
2021-12-27 10:51:03,797 INFO cluster.StandaloneSchedulerBackend: Granted executor ID app-20211227105103-0014/1 on hostPort <IP_ADDRESS>:36031 with 1 core(s), 27.0 GiB RAM
2021-12-27 10:51:03,797 INFO client.StandaloneAppClient$ClientEndpoint: Executor added: app-20211227105103-0014/2 on worker-20211226140545-<IP_ADDRESS>-36787 (<IP_ADDRESS>:36787) with 1 core(s)
2021-12-27 10:51:03,798 INFO cluster.StandaloneSchedulerBackend: Granted executor ID app-20211227105103-0014/2 on hostPort <IP_ADDRESS>:36787 with 1 core(s), 27.0 GiB RAM
2021-12-27 10:51:03,801 INFO util.Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 42229.
2021-12-27 10:51:03,801 INFO netty.NettyBlockTransferService: Server created on amax11:42229
2021-12-27 10:51:03,803 INFO storage.BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
2021-12-27 10:51:03,812 INFO storage.BlockManagerMaster: Registering BlockManager BlockManagerId(driver, amax11, 42229, None)
2021-12-27 10:51:03,815 INFO storage.BlockManagerMasterEndpoint: Registering block manager amax11:42229 with 14.2 GiB RAM, BlockManagerId(driver, amax11, 42229, None)
2021-12-27 10:51:03,818 INFO storage.BlockManagerMaster: Registered BlockManager BlockManagerId(driver, amax11, 42229, None)
2021-12-27 10:51:03,819 INFO storage.BlockManager: Initialized BlockManager: BlockManagerId(driver, amax11, 42229, None)
2021-12-27 10:51:03,822 INFO client.StandaloneAppClient$ClientEndpoint: Executor updated: app-20211227105103-0014/0 is now RUNNING
2021-12-27 10:51:03,822 INFO client.StandaloneAppClient$ClientEndpoint: Executor updated: app-20211227105103-0014/2 is now RUNNING
2021-12-27 10:51:03,823 INFO client.StandaloneAppClient$ClientEndpoint: Executor updated: app-20211227105103-0014/1 is now RUNNING
2021-12-27 10:51:03,934 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@5ab79e0c{/metrics/json,null,AVAILABLE,@Spark}
2021-12-27 10:51:04,320 INFO history.SingleEventLogFileWriter: Logging events to hdfs://amax11:8020/spark-sa-history/app-20211227105103-0014.inprogress
2021-12-27 10:51:04,521 INFO cluster.StandaloneSchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.0
args: Namespace(batch_size=64, cluster_size=3, epochs=3, export_dir='/home/amax/TOS/TensorFlowOnSpark/mnist_export', images_labels='/home/amax/TOS/TensorFlowOnSpark/data/mnist/csv/train', model_dir='/home/amax/TOS/TensorFlowOnSpark/mnist_model', tensorboard=False)
2021-12-27 10:51:04,712 INFO memory.MemoryStore: Block broadcast_0 stored as values in memory (estimated size 297.3 KiB, free 14.2 GiB)
2021-12-27 10:51:04,759 INFO memory.MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 27.4 KiB, free 14.2 GiB)
2021-12-27 10:51:04,762 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on amax11:42229 (size: 27.4 KiB, free: 14.2 GiB)
2021-12-27 10:51:04,765 INFO spark.SparkContext: Created broadcast 0 from textFile at NativeMethodAccessorImpl.java:0
2021-12-27 10:51:04,916 INFO (MainThread-11873) Reserving TFSparkNodes
2021-12-27 10:51:04,917 INFO (MainThread-11873) cluster_template: {'chief': [0], 'worker': [1, 2]}
2021-12-27 10:51:04,919 INFO (MainThread-11873) Reservation server binding to port 0
2021-12-27 10:51:04,919 INFO (MainThread-11873) listening for reservations at ('<IP_ADDRESS>', 37823)
2021-12-27 10:51:04,919 INFO (MainThread-11873) Starting TensorFlow on executors
2021-12-27 10:51:04,924 INFO resource.ResourceProfile: Default ResourceProfile created, executor resources: Map(cores -> name: cores, amount: 1, script: , vendor: , memory -> name: memory, amount: 27648, script: , vendor: ), task resources: Map(cpus -> name: cpus, amount: 1.0)
2021-12-27 10:51:04,928 INFO (MainThread-11873) Waiting for TFSparkNodes to start
2021-12-27 10:51:04,928 INFO (MainThread-11873) waiting for 3 reservations
2021-12-27 10:51:04,983 INFO spark.SparkContext: Starting job: foreachPartition at /home/amax/anaconda3/envs/tos_sa/lib/python3.6/site-packages/tensorflowonspark/TFCluster.py:327
2021-12-27 10:51:04,999 INFO scheduler.DAGScheduler: Got job 0 (foreachPartition at /home/amax/anaconda3/envs/tos_sa/lib/python3.6/site-packages/tensorflowonspark/TFCluster.py:327) with 3 output partitions
2021-12-27 10:51:04,999 INFO scheduler.DAGScheduler: Final stage: ResultStage 0 (foreachPartition at /home/amax/anaconda3/envs/tos_sa/lib/python3.6/site-packages/tensorflowonspark/TFCluster.py:327)
2021-12-27 10:51:05,000 INFO scheduler.DAGScheduler: Parents of final stage: List()
2021-12-27 10:51:05,000 INFO scheduler.DAGScheduler: Missing parents: List()
2021-12-27 10:51:05,003 INFO scheduler.DAGScheduler: Submitting ResultStage 0 (PythonRDD[3] at foreachPartition at /home/amax/anaconda3/envs/tos_sa/lib/python3.6/site-packages/tensorflowonspark/TFCluster.py:327), which has no missing parents
2021-12-27 10:51:05,012 INFO memory.MemoryStore: Block broadcast_1 stored as values in memory (estimated size 17.1 KiB, free 14.2 GiB)
2021-12-27 10:51:05,014 INFO memory.MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 11.7 KiB, free 14.2 GiB)
2021-12-27 10:51:05,015 INFO storage.BlockManagerInfo: Added broadcast_1_piece0 in memory on amax11:42229 (size: 11.7 KiB, free: 14.2 GiB)
2021-12-27 10:51:05,015 INFO spark.SparkContext: Created broadcast 1 from broadcast at DAGScheduler.scala:1200
2021-12-27 10:51:05,025 INFO scheduler.DAGScheduler: Submitting 3 missing tasks from ResultStage 0 (PythonRDD[3] at foreachPartition at /home/amax/anaconda3/envs/tos_sa/lib/python3.6/site-packages/tensorflowonspark/TFCluster.py:327) (first 15 tasks are for partitions Vector(0, 1, 2))
2021-12-27 10:51:05,026 INFO scheduler.TaskSchedulerImpl: Adding task set 0.0 with 3 tasks
2021-12-27 10:51:05,245 INFO cluster.CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(spark-client://Executor) (<IP_ADDRESS>:52680) with ID 2
2021-12-27 10:51:05,279 INFO cluster.CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(spark-client://Executor) (<IP_ADDRESS>:40504) with ID 1
2021-12-27 10:51:05,286 INFO cluster.CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(spark-client://Executor) (<IP_ADDRESS>:44508) with ID 0
2021-12-27 10:51:05,343 INFO storage.BlockManagerMasterEndpoint: Registering block manager <IP_ADDRESS>:33393 with 14.2 GiB RAM, BlockManagerId(2, <IP_ADDRESS>, 33393, None)
2021-12-27 10:51:05,361 INFO storage.BlockManagerMasterEndpoint: Registering block manager <IP_ADDRESS>:38801 with 14.2 GiB RAM, BlockManagerId(1, <IP_ADDRESS>, 38801, None)
2021-12-27 10:51:05,363 INFO storage.BlockManagerMasterEndpoint: Registering block manager <IP_ADDRESS>:35421 with 14.2 GiB RAM, BlockManagerId(0, <IP_ADDRESS>, 35421, None)
2021-12-27 10:51:05,391 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, <IP_ADDRESS>, executor 2, partition 0, PROCESS_LOCAL, 7337 bytes)
2021-12-27 10:51:05,398 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 0.0 (TID 1, <IP_ADDRESS>, executor 1, partition 1, PROCESS_LOCAL, 7337 bytes)
2021-12-27 10:51:05,399 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 0.0 (TID 2, <IP_ADDRESS>, executor 0, partition 2, PROCESS_LOCAL, 7337 bytes)
2021-12-27 10:51:05,559 INFO storage.BlockManagerInfo: Added broadcast_1_piece0 in memory on <IP_ADDRESS>:38801 (size: 11.7 KiB, free: 14.2 GiB)
2021-12-27 10:51:05,560 INFO storage.BlockManagerInfo: Added broadcast_1_piece0 in memory on <IP_ADDRESS>:35421 (size: 11.7 KiB, free: 14.2 GiB)
2021-12-27 10:51:05,566 INFO storage.BlockManagerInfo: Added broadcast_1_piece0 in memory on <IP_ADDRESS>:33393 (size: 11.7 KiB, free: 14.2 GiB)
2021-12-27 10:51:05,930 INFO (MainThread-11873) waiting for 3 reservations
2021-12-27 10:51:06,931 INFO (MainThread-11873) waiting for 3 reservations
2021-12-27 10:51:07,932 INFO (MainThread-11873) waiting for 3 reservations
2021-12-27 10:51:08,934 INFO (MainThread-11873) waiting for 3 reservations
2021-12-27 10:51:09,935 INFO (MainThread-11873) waiting for 3 reservations
2021-12-27 10:51:10,936 INFO (MainThread-11873) all reservations completed
2021-12-27 10:51:10,937 INFO (MainThread-11873) All TFSparkNodes started
2021-12-27 10:51:10,937 INFO (MainThread-11873) {'executor_id': 1, 'host': '<IP_ADDRESS>', 'job_name': 'worker', 'task_index': 0, 'port': 36945, 'tb_pid': 0, 'tb_port': 0, 'addr': '/tmp/pymp-8lelocpu/listener-c8uef51e', 'authkey': b'n/\x82\x08F\x94L\xb0\xb5T\xd0\xf4\xb6\n\xb2y'}
2021-12-27 10:51:10,937 INFO (MainThread-11873) {'executor_id': 2, 'host': '<IP_ADDRESS>', 'job_name': 'worker', 'task_index': 1, 'port': 45615, 'tb_pid': 0, 'tb_port': 0, 'addr': '/tmp/pymp-k4cka2ni/listener-um3u1iby', 'authkey': b'\x9e\xfa+\x93\xb2bFx\xa1\x84pQ\xba\xef=6'}
2021-12-27 10:51:10,937 INFO (MainThread-11873) {'executor_id': 0, 'host': '<IP_ADDRESS>', 'job_name': 'chief', 'task_index': 0, 'port': 44093, 'tb_pid': 0, 'tb_port': 0, 'addr': '/tmp/pymp-sevn96me/listener-coe66f9n', 'authkey': b'3\xd8\x85\xe7\x06UKa\x8b\xcb\xe6\x11\n\xa1)C'}
2021-12-27 10:51:10,937 INFO (MainThread-11873) Feeding training data
2021-12-27 10:51:11,023 INFO mapred.FileInputFormat: Total input files to process : 10
2021-12-27 10:51:11,061 INFO spark.SparkContext: Starting job: collect at PythonRDD.scala:168
2021-12-27 10:51:11,064 INFO scheduler.DAGScheduler: Got job 1 (collect at PythonRDD.scala:168) with 30 output partitions
2021-12-27 10:51:11,064 INFO scheduler.DAGScheduler: Final stage: ResultStage 1 (collect at PythonRDD.scala:168)
2021-12-27 10:51:11,064 INFO scheduler.DAGScheduler: Parents of final stage: List()
2021-12-27 10:51:11,065 INFO scheduler.DAGScheduler: Missing parents: List()
2021-12-27 10:51:11,066 INFO scheduler.DAGScheduler: Submitting ResultStage 1 (PythonRDD[6] at RDD at PythonRDD.scala:53), which has no missing parents
2021-12-27 10:51:11,078 INFO memory.MemoryStore: Block broadcast_2 stored as values in memory (estimated size 12.6 KiB, free 14.2 GiB)
2021-12-27 10:51:11,079 INFO memory.MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 7.3 KiB, free 14.2 GiB)
2021-12-27 10:51:11,080 INFO storage.BlockManagerInfo: Added broadcast_2_piece0 in memory on amax11:42229 (size: 7.3 KiB, free: 14.2 GiB)
2021-12-27 10:51:11,080 INFO spark.SparkContext: Created broadcast 2 from broadcast at DAGScheduler.scala:1200
2021-12-27 10:51:11,081 INFO scheduler.DAGScheduler: Submitting 30 missing tasks from ResultStage 1 (PythonRDD[6] at RDD at PythonRDD.scala:53) (first 15 tasks are for partitions Vector(0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14))
2021-12-27 10:51:11,081 INFO scheduler.TaskSchedulerImpl: Adding task set 1.0 with 30 tasks
2021-12-27 10:51:15,330 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 1.0 (TID 3, <IP_ADDRESS>, executor 2, partition 0, ANY, 7536 bytes)
2021-12-27 10:51:15,333 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 9956 ms on <IP_ADDRESS> (executor 2) (1/3)
2021-12-27 10:51:15,341 INFO python.PythonAccumulatorV2: Connected to AccumulatorServer at host: <IP_ADDRESS> port: 45655
2021-12-27 10:51:15,361 INFO storage.BlockManagerInfo: Added broadcast_2_piece0 in memory on <IP_ADDRESS>:33393 (size: 7.3 KiB, free: 14.2 GiB)
2021-12-27 10:51:15,387 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on <IP_ADDRESS>:33393 (size: 27.4 KiB, free: 14.2 GiB)
2021-12-27 10:51:15,650 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 1.0 (TID 4, <IP_ADDRESS>, executor 0, partition 1, ANY, 7536 bytes)
2021-12-27 10:51:15,650 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 0.0 (TID 2) in 10251 ms on <IP_ADDRESS> (executor 0) (2/3)
2021-12-27 10:51:15,691 INFO storage.BlockManagerInfo: Added broadcast_2_piece0 in memory on <IP_ADDRESS>:35421 (size: 7.3 KiB, free: 14.2 GiB)
2021-12-27 10:51:15,712 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 1.0 (TID 5, <IP_ADDRESS>, executor 1, partition 2, ANY, 7536 bytes)
2021-12-27 10:51:15,712 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 0.0 (TID 1) in 10314 ms on <IP_ADDRESS> (executor 1) (3/3)
2021-12-27 10:51:15,714 INFO scheduler.DAGScheduler: ResultStage 0 (foreachPartition at /home/amax/anaconda3/envs/tos_sa/lib/python3.6/site-packages/tensorflowonspark/TFCluster.py:327) finished in 10.703 s
2021-12-27 10:51:15,714 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool
2021-12-27 10:51:15,715 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on <IP_ADDRESS>:35421 (size: 27.4 KiB, free: 14.2 GiB)
2021-12-27 10:51:15,719 INFO scheduler.DAGScheduler: Job 0 is finished. Cancelling potential speculative or zombie tasks for this job
2021-12-27 10:51:15,719 INFO scheduler.TaskSchedulerImpl: Killing all running tasks in stage 0: Stage finished
2021-12-27 10:51:15,721 INFO scheduler.DAGScheduler: Job 0 finished: foreachPartition at /home/amax/anaconda3/envs/tos_sa/lib/python3.6/site-packages/tensorflowonspark/TFCluster.py:327, took 10.737965 s
2021-12-27 10:51:15,749 INFO storage.BlockManagerInfo: Added broadcast_2_piece0 in memory on <IP_ADDRESS>:38801 (size: 7.3 KiB, free: 14.2 GiB)
2021-12-27 10:51:15,779 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on <IP_ADDRESS>:38801 (size: 27.4 KiB, free: 14.2 GiB)
The logs from Spark UI:
Spark Executor Command: "/opt/module/jdk1.8.0_212/bin/java" "-cp" "/opt/module/spark-standalone/conf/:/opt/module/spark-standalone/jars/*" "-Xmx27648M" "-Dspark.driver.port=45175" "org.apache.spark.executor.CoarseGrainedExecutorBackend" "--driver-url" "spark://CoarseGrainedScheduler@amax11:45175" "--executor-id" "0" "--hostname" "<IP_ADDRESS>" "--cores" "1" "--app-id" "app-20211227105103-0014" "--worker-url" "spark://Worker@<IP_ADDRESS>:43273"
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
21/12/27 10:51:04 INFO CoarseGrainedExecutorBackend: Started daemon with process name: 11957@amax11
21/12/27 10:51:04 INFO SignalUtils: Registered signal handler for TERM
21/12/27 10:51:04 INFO SignalUtils: Registered signal handler for HUP
21/12/27 10:51:04 INFO SignalUtils: Registered signal handler for INT
21/12/27 10:51:04 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
21/12/27 10:51:04 INFO SecurityManager: Changing view acls to: amax
21/12/27 10:51:04 INFO SecurityManager: Changing modify acls to: amax
21/12/27 10:51:04 INFO SecurityManager: Changing view acls groups to:
21/12/27 10:51:04 INFO SecurityManager: Changing modify acls groups to:
21/12/27 10:51:04 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(amax); groups with view permissions: Set(); users  with modify permissions: Set(amax); groups with modify permissions: Set()
21/12/27 10:51:04 INFO TransportClientFactory: Successfully created connection to amax11/<IP_ADDRESS>:45175 after 54 ms (0 ms spent in bootstraps)
21/12/27 10:51:04 INFO SecurityManager: Changing view acls to: amax
21/12/27 10:51:04 INFO SecurityManager: Changing modify acls to: amax
21/12/27 10:51:04 INFO SecurityManager: Changing view acls groups to:
21/12/27 10:51:04 INFO SecurityManager: Changing modify acls groups to:
21/12/27 10:51:04 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(amax); groups with view permissions: Set(); users  with modify permissions: Set(amax); groups with modify permissions: Set()
21/12/27 10:51:05 INFO TransportClientFactory: Successfully created connection to amax11/<IP_ADDRESS>:45175 after 1 ms (0 ms spent in bootstraps)
21/12/27 10:51:05 INFO DiskBlockManager: Created local directory at /tmp/spark-66944227-f0bf-4829-b870-f5d85feae2cd/executor-2c04267d-7763-4e33-b90c-a6c1ead9f50b/blockmgr-6ac139bf-9f9c-496c-82de-a559a0202fad
21/12/27 10:51:05 INFO MemoryStore: MemoryStore started with capacity 14.2 GiB
21/12/27 10:51:05 INFO CoarseGrainedExecutorBackend: Connecting to driver: spark://CoarseGrainedScheduler@amax11:45175
21/12/27 10:51:05 INFO WorkerWatcher: Connecting to worker spark://Worker@<IP_ADDRESS>:43273
21/12/27 10:51:05 INFO TransportClientFactory: Successfully created connection to /<IP_ADDRESS>:43273 after 1 ms (0 ms spent in bootstraps)
21/12/27 10:51:05 INFO WorkerWatcher: Successfully connected to spark://Worker@<IP_ADDRESS>:43273
21/12/27 10:51:05 INFO ResourceUtils: ==============================================================
21/12/27 10:51:05 INFO ResourceUtils: Resources for spark.executor:
21/12/27 10:51:05 INFO ResourceUtils: ==============================================================
21/12/27 10:51:05 INFO CoarseGrainedExecutorBackend: Successfully registered with driver
21/12/27 10:51:05 INFO Executor: Starting executor ID 0 on host <IP_ADDRESS>
21/12/27 10:51:05 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 35421.
21/12/27 10:51:05 INFO NettyBlockTransferService: Server created on <IP_ADDRESS>:35421
21/12/27 10:51:05 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
21/12/27 10:51:05 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(0, <IP_ADDRESS>, 35421, None)
21/12/27 10:51:05 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(0, <IP_ADDRESS>, 35421, None)
21/12/27 10:51:05 INFO BlockManager: Initialized BlockManager: BlockManagerId(0, <IP_ADDRESS>, 35421, None)
21/12/27 10:51:05 INFO CoarseGrainedExecutorBackend: Got assigned task 2
21/12/27 10:51:05 INFO Executor: Running task 2.0 in stage 0.0 (TID 2)
21/12/27 10:51:05 INFO TorrentBroadcast: Started reading broadcast variable 1 with 1 pieces (estimated total size 4.0 MiB)
21/12/27 10:51:05 INFO TransportClientFactory: Successfully created connection to amax11/<IP_ADDRESS>:42229 after 2 ms (0 ms spent in bootstraps)
21/12/27 10:51:05 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 11.7 KiB, free 14.2 GiB)
21/12/27 10:51:05 INFO TorrentBroadcast: Reading broadcast variable 1 took 90 ms
21/12/27 10:51:05 INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 17.1 KiB, free 14.2 GiB)
2021-12-27 10:51:10,558 INFO (MainThread-12074) Available GPUs: ['0']
2021-12-27 10:51:10,559 INFO (MainThread-12074) Proposed GPUs: ['0']
2021-12-27 10:51:10,559 INFO (MainThread-12074) Requested 1 GPU(s), setting CUDA_VISIBLE_DEVICES=0
2021-12-27 10:51:10,578 INFO (MainThread-12074) connected to server at ('<IP_ADDRESS>', 37823)
2021-12-27 10:51:10,578 INFO (MainThread-12074) TFSparkNode.reserve: {'executor_id': 2, 'host': '<IP_ADDRESS>', 'job_name': 'worker', 'task_index': 1, 'port': 45615, 'tb_pid': 0, 'tb_port': 0, 'addr': '/tmp/pymp-k4cka2ni/listener-um3u1iby', 'authkey': b'\x9e\xfa+\x93\xb2bFx\xa1\x84pQ\xba\xef=6'}
2021-12-27 10:51:12,582 INFO (MainThread-12074) node: {'executor_id': 0, 'host': '<IP_ADDRESS>', 'job_name': 'chief', 'task_index': 0, 'port': 44093, 'tb_pid': 0, 'tb_port': 0, 'addr': '/tmp/pymp-sevn96me/listener-coe66f9n', 'authkey': b'3\xd8\x85\xe7\x06UKa\x8b\xcb\xe6\x11\n\xa1)C'}
2021-12-27 10:51:12,582 INFO (MainThread-12074) node: {'executor_id': 1, 'host': '<IP_ADDRESS>', 'job_name': 'worker', 'task_index': 0, 'port': 36945, 'tb_pid': 0, 'tb_port': 0, 'addr': '/tmp/pymp-8lelocpu/listener-c8uef51e', 'authkey': b'n/\x82\x08F\x94L\xb0\xb5T\xd0\xf4\xb6\n\xb2y'}
2021-12-27 10:51:12,582 INFO (MainThread-12074) node: {'executor_id': 2, 'host': '<IP_ADDRESS>', 'job_name': 'worker', 'task_index': 1, 'port': 45615, 'tb_pid': 0, 'tb_port': 0, 'addr': '/tmp/pymp-k4cka2ni/listener-um3u1iby', 'authkey': b'\x9e\xfa+\x93\xb2bFx\xa1\x84pQ\xba\xef=6'}
2021-12-27 10:51:12,582 INFO (MainThread-12074) export TF_CONFIG: {"cluster": {"chief": ["<IP_ADDRESS>:44093"], "worker": ["<IP_ADDRESS>:36945", "<IP_ADDRESS>:45615"]}, "task": {"type": "worker", "index": 1}, "environment": "cloud"}
2021-12-27 10:51:15,616 INFO (MainThread-12074) Available GPUs: ['0']
2021-12-27 10:51:15,616 INFO (MainThread-12074) Proposed GPUs: ['0']
2021-12-27 10:51:15,616 INFO (MainThread-12074) Requested 1 GPU(s), setting CUDA_VISIBLE_DEVICES=0
2021-12-27 10:51:15,616 INFO (MainThread-12074) Starting TensorFlow worker:1 as worker on cluster node 2 on background process
21/12/27 10:51:15 INFO PythonRunner: Times: total = 9778, boot = 316, init = 1021, finish = 8441
2021-12-27 10:51:15,627 WARNING (MainThread-12152) From /home/amax/TOS/TensorFlowOnSpark/examples/mnist/keras/mnist_spark.py:11: _CollectiveAllReduceStrategyExperimental.init (from tensorflow.python.distribute.collective_all_reduce_strategy) is deprecated and will be removed in a future version.
Instructions for updating:
use distribute.MultiWorkerMirroredStrategy instead
21/12/27 10:51:15 INFO Executor: Finished task 2.0 in stage 0.0 (TID 2). 1549 bytes result sent to driver
21/12/27 10:51:15 INFO CoarseGrainedExecutorBackend: Got assigned task 4
21/12/27 10:51:15 INFO Executor: Running task 1.0 in stage 1.0 (TID 4)
21/12/27 10:51:15 INFO TorrentBroadcast: Started reading broadcast variable 2 with 1 pieces (estimated total size 4.0 MiB)
21/12/27 10:51:15 INFO TransportClientFactory: Successfully created connection to /<IP_ADDRESS>:33393 after 1 ms (0 ms spent in bootstraps)
21/12/27 10:51:15 INFO MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 7.3 KiB, free 14.2 GiB)
21/12/27 10:51:15 INFO TorrentBroadcast: Reading broadcast variable 2 took 25 ms
21/12/27 10:51:15 INFO MemoryStore: Block broadcast_2 stored as values in memory (estimated size 12.6 KiB, free 14.2 GiB)
21/12/27 10:51:15 INFO HadoopRDD: Input split: hdfs://amax11:8020/home/amax/TOS/TensorFlowOnSpark/data/mnist/csv/train/part-00001:0+11232549
21/12/27 10:51:15 INFO TorrentBroadcast: Started reading broadcast variable 0 with 1 pieces (estimated total size 4.0 MiB)
21/12/27 10:51:15 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 27.4 KiB, free 14.2 GiB)
21/12/27 10:51:15 INFO TorrentBroadcast: Reading broadcast variable 0 took 9 ms
21/12/27 10:51:15 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 426.4 KiB, free 14.2 GiB)
2021-12-27 10:51:16.542093: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2021-12-27 10:51:16.980995: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1510] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 8107 MB memory:  -> device: 0, name: GeForce RTX 3080, pci bus id: 0000:65:00.0, compute capability: 8.6
2021-12-27 10:51:16.989052: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1510] Created device /job:worker/replica:0/task:1/device:GPU:0 with 8107 MB memory:  -> device: 0, name: GeForce RTX 3080, pci bus id: 0000:65:00.0, compute capability: 8.6
2021-12-27 10:51:16.994323: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:272] Initialize GrpcChannelCache for job chief -> {0 -> <IP_ADDRESS>:44093}
2021-12-27 10:51:16.994344: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:272] Initialize GrpcChannelCache for job worker -> {0 -> <IP_ADDRESS>:36945, 1 -> <IP_ADDRESS>:45615}
2021-12-27 10:51:16.995505: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:427] Started server with target: grpc://<IP_ADDRESS>:45615
2021-12-27 10:51:17,003 INFO (MainThread-12152) Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:GPU:0']
2021-12-27 10:51:17,104 INFO (MainThread-12152) Waiting for the cluster, timeout = inf
2021-12-27 10:51:17,194 INFO (MainThread-12169) Connected to TFSparkNode.mgr on <IP_ADDRESS>, executor=2, state='running'
2021-12-27 10:51:17,201 INFO (MainThread-12169) mgr.state='running'
2021-12-27 10:51:17,201 INFO (MainThread-12169) Feeding partition <itertools.chain object at 0x7f8091c66d30> into input queue <multiprocessing.queues.JoinableQueue object at 0x7f7fe8a757f0>
21/12/27 10:51:17 INFO PythonRunner: Times: total = 772, boot = -382, init = 438, finish = 716
Spark Submit Command Line:
${SPARK_HOME}/bin/spark-submit
--master ${MASTER}
--conf spark.cores.max=${TOTAL_CORES}
--conf spark.task.cpus=${CORES_PER_WORKER}
--executor-memory 27G
${TFoS_HOME}/examples/mnist/keras/mnist_spark.py
--cluster_size ${SPARK_WORKER_INSTANCES}
--images_labels ${TFoS_HOME}/data/mnist/csv/train
--model_dir ${TFoS_HOME}/mnist_model
--export_dir ${TFoS_HOME}/mnist_export

My guess is that your model_dir needs to fully-specify the HDFS path, e.g. hdfs://default/...
Note that the example uses spark-local mode (for simplicity).


RSA_Algorithm_Script added

Description
RSA_Algorithm is added
#2536
Type of change
Please delete options that are not relevant.

[x] New feature (non-breaking change which adds functionality)

Checklist:

[x] My code follows the style guidelines(Clean Code) of this project
[x] I have performed a self-review of my own code
[x] I have commented my code, particularly in hard-to-understand areas
[x] I have created a helpful and easy to understand README.md
[x] My documentation follows Template for README.md
[ ] I have added the project meta data in the PR template.

Project Metadata
none
Category:

[ ] Calculators
[ ] AI/ML
[ ] Scrappers
[ ] Social_Media
[ ] PDF
[ ] Image_Processing
[ ] Video_Processing
[ ] Games
[ ] Networking
[ ] OS_Utilities
[x] Automation
[ ] Cryptography
[ ] Computer_Vision
[ ] Fun
[ ] Others

can you please review this pr  @avinashkranjan


Update ci-triggers5.md

Update tips about wildcard use cases according to latest release notes regarding wildcards in trigger paths: https://docs.microsoft.com/en-us/azure/devops/release-notes/2021/sprint-192-update#support-for-wild-cards-in-path-filters

@PavloAndriiesh : Thanks for your contribution! The author(s) have been notified to review your proposed change.

@PavloAndriiesh : Thanks for your contribution! The author(s) have been notified to review your proposed change.

#label:"aq-pr-triaged"

@ktoliver : Thanks for your contribution! The author(s) have been notified to review your proposed change.


[WIP] Add getter of minimal attributes necessary for pulse simulation in FakeBackendV2

Summary
Exceptionally add getter function of two custom attributes hamiltonian and u_channel_lo to FakeBackendV2. Those attributes are necessary for pulse simulation. They were in configuration() in BackendV1 but they disappear in BackendV2.
Details and comments
Currently, FakeManila().configuration().hamiltonian (u_channel_lo) returns Hamiltonian string (list of u channels info) while the counterpart FakeManilaV2().hamiltonian (u_channel_lo) fails with AttributeError. This commit enables to get those two attributes without failing by adding FakeBackendV2.__getattr__() function.

With Aer having removed pulse simulation in Aer 0.13 and the deprecation/removal of pulse simulation from the fake backends in #11439 and #11442, I think this PR is obsolete now.  I'll close it, but please feel free to re-open if there's more to discuss.


Reset Scoreboard

I'd like to reset my scoreboard. I removed the app from Slack and re-installed it, but the old scores remained.

I am sorry, the application has no real way to reset the scores.


Error:/usr/local/include/ObjectiveC2/objc/runtime.h:146:1: error: unknown type name 'Method_t'; did you mean 'Method'?

/usr/local/include/ObjectiveC2/objc/runtime.h:146:1: error: unknown type name 'Method_t'; did you mean 'Method'?
Centos
command:
clang -fblocks -fobjc-nonfragile-abi main.m -o main

I think that you have filed this bug in the wrong place.  If you have ObjectiveC2 headers then you are using GNUstep-base's compatibility layer for old versions of the GCC Objective-C runtime.  This should not be used with libobjc2.  If you are using libobjc2, then you have misconfigured GNUstep-base.


Abnormally high CPU usage

Continuing from https://github.com/pacmancoder/rustzx/issues/30#issuecomment-233913754, I've retested on a 2.2 GHz x86 Athlon machine (about 3x higher single-core performance + Nvidia OpenGL 1.5 + hardware sound mixing) and once a tape is loaded, it starts using 100% CPU right away (focus makes no difference). The sound is choppy too (probably slightly less than on ARM).
Even an empty "Sinclair Research" screen is using 100% cycles.
EDIT:
For reference, fuse-sdl (no scaling, SDL 1.x) normally uses 6-12% of the same ARM CPU and pulseaudio another 3-5%.

Hmmmm. Interesting. Tested on i3-3110M 2.4 GHz. 8% cpu, 5-6 % without sound. Weird. Maybe I can improve speed with faster unsafe memory copy functions when loading texture. That's might help, I think. From your profiling info I can conclude some things:

CPU mod is well-optimized.

but i must optimize registers struct and inline all its methods

SDL texture update eats A LOT  of time, i must optimize this.
I must improve canvas [emulated screen], it uses a lot of time.

Btw, @petevine , how to obtain such profiling info with rust as you get?

I originally intended to use cargo profiler +  callgrind with nicely demangled function names but it was too slow. So I used  operf on a release binary compiled with -g and annotated with:
opreport -g --symbols

Edited the original post to include a baseline.

Ditto here.


[ascend]feat: support kv int8

Thanks for your contribution and we appreciate it a lot. The following instructions would make your pull request more healthy and more easily receiving feedbacks. If you do not understand some items, don't worry, just make the pull request and seek help from maintainers.
Motivation
This PR support kv int8 on ascend device.
Modification
This PR modified some paras in CacheEngine, context in model_forward，and op_backend of ascend.
BC-breaking (Optional)
Does the modification introduce changes that break the backward-compatibility of the downstream repositories?
If so, please describe how it breaks the compatibility and how the downstream projects should modify their code to keep compatibility with this PR.
Use cases (Optional)
If this PR introduces a new feature, it is better to list some use cases here, and update the documentation.
Checklist

Pre-commit or other linting tools are used to fix the potential lint issues.
The modification is covered by complete unit tests. If not, please add more unit tests to ensure the correctness.
If the modification has a dependency on downstream projects of a newer version, this PR should be tested with all supported versions of downstream projects.
The documentation has been modified accordingly, like docstring or example tutorials.

what is ASCEND_QUANT_RECORD_FILE

what is ASCEND_QUANT_RECORD_FILE

ascend only supports offline kv-quant。 ASCEND_QUANT_RECORD_FILE is the file contains all kv scales and offsets.
please refer this doc：https://github.com/DeepLink-org/dlinfer/blob/main/docs/quant/ascend_kv_quant.md

we will update docs for this pr in another pr

why not put the doc in this PR?

doc updated


SAFE Browser appears not to support Promise.finally()

Prerequisites

[ ] Using yarn
[x] Using an up-to-date master branch
[ ] Link to stacktrace in a Gist (for bugs)

Expected Behavior

Major browsers support Promise.finally() so SAFE Browser should also, as in:
safeApp.auth.loginFromUri(authUri).then(() => {
      console.log('loginFromUri')
  }).catch((e) => {
      console.log('catch: ', e)
  }).finally(() => {
      console.log('finally!!!')
  })

Current Behavior

Using Promise.finally() such as in the above example results in an error:
Uncaught (in promise) TypeError: safeApp.auth.loginFromUri(...).then(...).catch(...).finally is not a function

UPDATE: as a test I have tried this with the completed SAFE Web App Tutorial example (in Vue), again chaging the safeApp.auth.loginFromUri(authUri) line to the above code fragment, and I get the same error.
So the issue is not restricted to my version of the tutorial implemented using svelte, and suggests it is a problem with the Promise implementation exposed to web apps by SAFE Browser.
Possible Solution

This might be because the browser JavaScript engine doesn't support Promise.finally() or another possible cause is if a polyfill (perhaps from a dependency) is overwriting the browser implementation.
Your Environment

Node version : 10
Version or Branch used : 0.13 dev build
Operating System and version : Ubuntu 18.10
Link to your project : https://github.com/theWebalyst/safetutorial-svelte
I've modified the file safe-api.js to use the code fragment above instead of the following line:

https://github.com/theWebalyst/safetutorial-svelte/blob/master/src/safe-api.js#L15

UPDATE: as a test I have tried this with the completed SAFE Web App Tutorial example (in Vue), again chaging the safeApp.auth.loginFromUri(authUri) line to the above code fragment, and I get the same error.
So the issue is not restricted to my version of the tutorial implemented using svelte, and suggests it is a problem with the Promise implementation exposed to web apps by SAFE Browser.

UPDATE: I'm a bit puzzled by this issue because Promise.finally() has been supported since at least Chrome v63, and if I understand correctly, v4.x of Electron which SAFE Browser v0.14 uses, has more recent Chromium than v63.

Closing because I believe this is fixed in SAFE Browser 0.15.1 (presumably by the update of electron).

Thanks @theWebalyst !


Add a Gitter chat badge to README.md

Fewlaps/quitnow-email-suggester now has a Chat Room on Gitter
@rocboronat has just created a chat room. You can visit it here: https://gitter.im/Fewlaps/quitnow-email-suggester.
This pull-request adds this badge to your README.md:

If my aim is a little off, please let me know.
Happy chatting.
PS: Click here if you would prefer not to receive automatic pull-requests from Gitter in future.

@gitter-badger it has been a little spammy, guys...


Create README.md file

Just created first version for README.md file.

Actually, there was a typo in the file name 🙃


vault.bitwarden.com login broken

Logging in to Bitwarden Cloud has started to return 503 errors.

This was because of a server bug and some TLS fingerprinting. Switching to the separate API and identity hosts in version 0.8.0 seems to have fixed the issue.
More info here: https://www.lkoskela.fi/blog/2023-09-02-wden-windows-503/


Does anything in current P4_16 spec prevent an extern from adding entries to a table?

I will think about where to record an answer to this later, for future reference, but for now I wanted to double-check my understanding.
If one wanted to create an extern function for an architecture that could, from within the P4 program, cause entries to be added to a P4 table, I do not believe that this would violate anything in the P4_16 language spec that I can see.
Some justification for this belief:
Section 6.7.1 "Justification", explaining what extern functions cannot do, pretty much lists the only things that an extern CANNOT do is: read or write P4 program variables, except via parameters to the extern function call.
It explicitly does say that externs can have these kinds of side effects (in version 1.2.1 of the spec, in case this text has changed across spec versions):
"In general, extern functions are arbitrarily powerful: they can store information in global storage, spawn separate threads, “collude” with each other to share information — but they cannot access any variable in a P4 program. With copy-in/copy-out semantics the compiler can still reason about P4 programs that invoke extern functions."
It does not explicitly say "an extern can add entries to a table", but I it seems unwise to attempt to explicitly list everything possible for a construct that is "arbitrarily powerful".
In particular, adding entries to a table via an extern call would not modify any P4 program variables, nor read them, except any passed as parameters to the extern call.
Happy to hear of any dissenting views on this, in case I have missed something.

Note: The questions raised in this issue do not say anything about whether a P4 target should be required to support such a feature.  That is properly left to particular target implementations to decide.  There are devices that can add entries to tables without involving the control plane for every such addition.
The point of the question is whether the P4 language spec would need to be modified in order to make this legal according to the spec.

@jafingerhut -- I think your understanding is correct. I do not see anything in the spec that would prohibit that and I also think that it was intentional.

Indeed, in the ebpf back-end we allow programs to write directly to tables.

@mbudiu-vmw Out of curiosity for the API you have created for this capability, do you happen to have an ebpf test program handy that exercises this feature?

You may be unsatisfied with this answer, but counters in ebpf are implemented as regular tables, but which are mutated by the dataplane directly. In ebpf_model.p4 the CounterArray extern has 'increment' and 'add' methods. So we have not implemented mutation for proper P4 tables, but CounterArray tables use really the same underlying construct. The count_ebpf.p4 program from p4_16_samples uses increment, and there is a count_ebpf.stf file.

It wasn't what I was hoping for, which was an implementation of a P4 data plane adding new entries to a P4 table, but I do appreciate knowing the differences between what you have done, and that.  Thanks.

There would be no difficulty to create an
extern TableWriter<K,V> {
   TableWriter(string table);
   void insert(k v, V v);
}

that is attached to a specific table (specified by name; let's assume that has exact keys), and which can insert an entry at a specified key. You would have to somehow encode in the value the data describing an action and its directionless arguments; perhaps you will also use a string for the action name. Both K and V will in general be tuple types. At runtime, calling 'insert' would overwrite the entry with the specified key with the specified tuple.  In the ebpf back-end the implementation of this method would be exactly the same as for CounterArray.insert.
Of course, this has no type-checking as written, and in general, if you can have multiple actions, you will need one of the unions that you proposed for the V values.

Or, perhaps to accommodate multiple actions the signature can be:
extern TableWriter<K> {
    TableWriter(string table);
    void insert<V>(K v, V v);
}

This way you can have multiple insert methods with different signatures.

  I am not opposed to the more general one that you proposed here https://github.com/p4lang/p4-spec/issues/889#issuecomment-683137478 but the restricted variant is one that I know has multiple use cases, and I've seen multiple high speed data planes with this restricted capability.
The common case is enough to implement a feature like Netflow, where the only time that you ever want to add an entry to a table from the data plane is because you just did an apply() call on the table and got a miss.  You want future packets with the same key to get a hit.
For this use case, a restricted insert operation that is only allowed to be called from the default action of a table is enough.  Besides being easier for a target device to implement, it also suggests that the target table of the insert can be understood from the context -- it must be the table whose default_action is being executed right now.
Also, the key one wants to insert is the key that was just looked up and not found.  This suggests that we don't need to provide a key as a parameter, because the key would be understood from the context.  If this seems to violate the restriction that externs cannot read P4 variables that are not given as parameters, I think one could argue that the extern can access the key that was just looked up as internal state of the architecture.  To me, this looks similar to how action selector externs like those in v1model, PSA, and other architectures can access the values of table search keys with match_kind selector.
So, the only parameters needed for a restricted insert are an action name, and values for the action's directionless parameters.  As you point out, a string for the action name doesn't require anything new in the language spec.
It seems useful for readability to allow a struct type to specify the directionless action parameters, as in the code snippet below that uses a structure literal expression with a type cast at its beginning.
// This code snippet looks like it requires nothing new in the P4_16 language spec
action t1_act1 (bit<8> p1, bit<16> p2) {
     // code for action here (details not relevant for example)
}

struct t1_act1_params_t {
    bit<8> p1;
    bit<16> p2;
}

action t1_missed () {
    insert("t1_act1", (t1_act1_params_t) {p1 = 17, p2 = 35});
}

If that is reasonable, we could consider the convenience of some kind of syntactic sugar that creates a struct type like t1_act1_params_t from the definition of action t1_act1.  The field names and types would be the same as the parameter names and types of the directionless parameters of the action.  I don't have good proposed syntax for such a thing, but if the syntax ACTION_DATA(t1_act1) represented a type like t1_act1_params_t above, we could write this:
// This code only requires one new thing in the language spec -- a definition for ACTION_DATA(t1_act1)
action t1_act1 (bit<8> p1, bit<16> p2) {
     // code for action here (details not relevant for example)
}

action t1_missed () {
    insert("t1_act1", (ACTION_DATA(t1_act1)) {p1 = 17, p2 = 35});
}

For a more general insert operation that also let you specify the key to insert, something perhaps called KEY_SPEC(table_name) could be a struct type auto-generated from the table's key property.  I don't have any known use cases for this generality, though -- just tossing that idea out there in case someone else would like to propose a definition for such a thing.
A crazy idea: allow something that looks more like an action call as the parameter to insert.
// This code is the most far-out proposal of the bunch, and not needed, given the more conservative approaches above
action t1_act1 (bit<8> p1, bit<16> p2) {
     // code for action here (details not relevant for example)
}

action t1_missed () {
    insert(t1_act1(p1 = 17, p2 = 35));
}

That requires some brand new type in the P4_16 language as the type of the expression t1_act1(p1 = 17, p2 = 35).  That sounds like it is low benefit relative to the time required to define it, since I do not know any use cases for it other than this insert extern function.

@jafingerhut -- I would assume that you want the extern not only to insert the entry, but also to execute the specified action with that specified action data, correct? Or would you like to split these into two independent functionalities?

@vgurevich - I am still double-checking details, but it might be important in some use cases to have different behavior in each of these cases:
(a) miss, and add succeeded in adding a new entry
(b) miss, and add failed to add a new entry
(c) hit on an already-existing entry

@jafingerhut -- my question was mostly about (a), i.e. do we assume that the action specified in the extern would also be executed for the packet and not executed for in case of (b)? This will require the extern to know the outcome of a pretty complex operation before you make a decision to execute the action, but should still be doable (maybe with some restrictions), because I would also state that externs can probably execute somewhat asynchronously from the rest of the data plane program.
Also, is the assumption that aging will be done by the control plane only, you you would also want something like
   idle_timeout = delete(); /* Another extern that takes the key as an implicit parameter */

In my opinion, everything you are suggesting should be allowed, especially given the fact that the control plane is allowed to manipulate the stateful objects (tables included) completely asynchronously from the data plane. Thus, allowing the data plane to do it  should not open any holes, I think.

Our "p4++" proposal asks for the ability of P4 programs to mutate arbitrary control-plane tables.
I believe that the right way to do this is to finalize the "Safe unions" proposal that @jafingerhut wrote, and then to have each table declaration introduce automatically a safe union type that contains a variant for each action. Then your insert function can use a literal value of this union type.
I have to think a bit about actions that have other non-directionless arguments too.

In the interest of tidying up the set of active issues on the P4 specification repository, I'm marking this as "stalled" and closing it. Of course, we can always re-open it in the future if there is interest in resurrecting it.


[RFC] Easy deployment on any Kubernetes cluster

Status: Open for comments

Need
Today there are instructions on how to deploy Backstage locally but deploying it to a Kubernetes cluster requires building Docker images and Kubernetes manifests.
Scope
Managing cloud specific resources (like managed databases) is out of scope for this proposal.
Proposal
It would be nice to have a way to deploy it to any Kubernetes cluster regardless of its platform. One way to do that is to use Helm which is widely used and graduated in CNCF.
To deploy it one would need to install helm locally and run:
helm repo add spotify https://charts.spotify.com
helm install backstage-demo spotify/backstage

or for a more customized installation:
helm install -f backstage-prod.yaml backstage-prod spotify/backstage

Ideally the charts:

work on all Kubernetes clusters
can work with zero extra configuration (i.e. they have reasonable defaults, auto-generated credentials and certificates) but also are flexible enough for production deployments
can also include and install the sub-charts for the dependencies like PostgreSQL
should also admit configuration for Backstage to connect to an existing Postgres DB
would live in ~a Chart repository hosted by Spotify and published in Helm Hub~ (edit) the Backstage repo under contrib/helm (see comment below)

Additionally:

on each Backstage release new Docker images are built and published
there is good documentation on how to configure things
there are tests for the charts

Alternatives
Terraform, or Ansible could be alternatives to Helm charts, but these require extra infrastructure.
Kustomize or Ksonnet with plane Kubernetes manifests are other alternatives that let the user customize plain Kubernetes manifests, but we believe Helm is more mature and widely used than these.

Looks like there is related conversation in https://github.com/spotify/backstage/issues/2540.

We (Roadie) and more specifically, Martina, are prepared to contribute this work. We already have a prototype working which creates the PG instance in kubernetes and configures Backstage to talk to it over automatically generated certs.
There are some things we will need from Spotify in order to achieve the interface presented above.

A place to store Docker images. e.g. A Spotify controlled Docker Hub repo and secrets to push images.
a GitHub repo in the Spotify GitHub org to act as a Helm chart index. As described here.
A DNS record (CNAME) pointing https://charts.spotify.com at that GitHub repo.

Can't wait to hear some opinions.

Another related issue: https://github.com/spotify/backstage/issues/373

@dtuite @martina-if what's your thinking around what would be part of the docker images you want to store?
Regarding helm charts, we could also rely on raw GitHub URLs for now right? Same question here though, what would the helm charts actually include?

@Rugvip

Regarding helm charts, we could also rely on raw GitHub URLs for now right?

You mean like this?
helm repo add roadie https://raw.githubusercontent.com/RoadieHQ/helm-charts/master/index.yaml

I expect that would work. I haven't tested it yet.

@dtuite yeah that, to get started anyway

There are some things we will need from Spotify in order to achieve the interface presented above.

@dtuite please consider that Backstage is being donated to the CNCF and that we would want to move away from Spotify dependencies.

what's your thinking around what would be part of the docker images you want to store?

Probably an image for each piece: backend, frontend and lighthouse plugin to start with.

what would the helm charts actually include?

deployment, service, configmaps, etc. for Backstage backend
deployment, service, configMap, etc for Backstage frontend
deployment, service, etc. for lighthouse backend plugin
ingress and optional auto generated certificate
optional postgresql dependency. The alternative would be to configure the connection to an existing db.
secrets to store auto-generated credentials and certificates for postgresql. (This would be optional and values can be provided by the user).

Regarding helm charts, we could also rely on raw GitHub URLs for now right?

Hosting the charts with Github pages should be quite simple. Do you think this could be a good option here?

@martina-if What is the purpose of the backend and frontend images? is to provide an up-to-date demo deployment of the example app in this repo? If the goal is for actual deployment and use by organizations, is the plan to wrap them up in additional layers to make it possible to select which plugins to include?

@Rugvip I think, initially at least, yeah, they would provide a demo deployment of the example app. I guess then each company can use these charts as a starting point for their own charts for their own backstage app.
I haven't looked into customizing which plugins to include. What do you think we should do here? Perhaps limit the scope to a demo deployment?

Regarding hosting the charts. Given what stefan mentioned about Backstage being donated to CNCF, it's probably better that the charts live inside the backstage repo under charts/backstage, for example. As an example, flux and ArgoCD do it the same way.

possibly contrib/helm?

@stefanalund @Rugvip A couple more details regarding the potential dependencies of a demo installation:

ingress would be done through ingress-nginx. This means that for the demo setup to work we would require this controller to exist in the cluster.

DNS: I think there should be a way to deploy the demo backstage that doesn't require the user to create DNS entries. That will be relatively easy after this is merged. There should also be an option to use it with a domain. We could add instructions on how to use ExternalDNS to make it easier.

regarding the different certificates and CAs I think the demo deployment could work with auto-generated self-signed certificates. And a more advanced setup with cert-manager could also be set up as an opt-in, since it requires that the controller is already installed in the cluster.

What are your thoughts? I have a prototype that I could clean up and submit as a PR if you think it would be valuable.

@martina-if Those things aren't very Backstage specific, so it may be worth finding an external guide to base the entire thing on and mainly reference that. It's really just a single (or two) containers with external access right?
Nginx ingress + ExternalDNS + and cert-manager all seem like sane choices, although an option could be to use Traefik for ingress as it has native support for Let's Encrypt, making the setup slightly simpler. In the end the choices of the surrounding infra should ofc not be done based on any requirements from Backstage itself, but from the wider organization that it's deployed in.
Once we start considering splitting the backend up into individual pieces it becomes a bit more interesting though, but that requires some more work before we have a recommended way to do that imo.

That sounds like approval to me! 😄

Well yes xP

@martina-if this can probably be closed now right?

@martina-if
I think you can go next step.
User needs doker images and helm chart continuously updated.


Legibility of notebook names

Right now, with the custom icons, the legibility of the icon is not optimal. It is maybe necessary to separate the name from the icon, or redesign them to increase legibility.

I like them. It's your own library so you have a knowlegde what is there and which notebook contains which informations. So you don't really have to read the name every time you want to open a notebook

I was thinking of doing something more like evernote, where the name is displayed horizontally instead of vertically. I think it is really not optimal having to turn your neck to read...
Even when you know your library, having a more legible way of looking at it would be nice. Are you opposed to horizontal names?

no no, horizontal names would be fine, too. If you like to draw new images, please do that. I just don't see an issue with the way it is now

Still searching for good user interfaces for this. I found this kudos file manager, which has a nice material design.
We could imagine having the list of folders on the left, with, on top, the home (which would be clickable, to be changed into whatever other home folder we want). Not sure about subfolders, though.
Then on the right, instead of the list in the link, we could keep a file/notebook format, like the evernote previously mentionned, or something more standard.

Yes, putting the folders into a list on the left sounds good


Enable eslint react/no-invalid-html-attribute rule

[x] I have followed (at least) the PR section of the contributing guide.

I don't see how it's useful to enable it if it's unable to catch typos like in https://github.com/mui/material-ui/commit/3f0fa4a45190c57736a55418002fbc9362edd1c6. So I am closing this PR.


Fix two bugs: macOS MMseqs2 version and integer contig names

macOS MMseqs2 version
Currently, Dnaapler checks for an MMseqs2 version string in the format major.minor. However, the macOS MMseqs2 v13.45111 binary (downloaded from MMseqs2 release page) reports its version as a hash (45111b641859ed0ddd875b94d6fd1aef1a675b7e). This causes Dnaapler's version check to fail.
Fix: Updated the version-check logic to support hash-based version numbers.
Integer contig names
When running Dnaapler on an Autocycler assembly with integer-named sequences, the following issue arose:
filtered_df = MMseqs2_df[MMseqs2_df["qseqid"] == short_contig]

In this case, short_contig was a string, but the qseqid column in the MMseqs2 results dataframe was inferred as an integer type. This type mismatch caused the filtered dataframe to always be empty.
Fix: Enforced the qseqid column to always load as a string (object type) when reading MMseqs2 results with Pandas.

I tested the results on my Mac using an Autocycler assembly, and it all worked well! But more thorough testing may be warranted if you've got some varied test cases :smile:

Legend thank you @rrwick ! - this would have been a pre v1.0.0 bug too


*ngIf disables dropdown functionality.

Hi @rubyboy,
I have the following code:

  dropdown item

    toggle

        dropdownarrow_drop_down

public drop: boolean = true;

toggle() {
  this.drop = !this.drop;
}

The dropdown works fine until you toggle it off and on.
Then it stops working. It seems it's not re-initializing the directive.
Maybe I'm missing something.

@lbertenasco
I have run into the same issue. Were you able to find a solution?

@lbertenasco @BruthaVoodoo , I think this is happening because the element gets destroyed and recreated when using ngIf and MaterializeCSS needs to be told about it.
See below a workaround. I'll try to figure out a good way to support that internally, but hopefully the workaround should get you going.
@Component({
    selector: "dropdown",
    template: `

     Drop Me!

one

two

three

     toggle
    `
})
export class Dropdown {
    public drop: boolean = true;
    dropdownActions = new EventEmitter<string|MaterializeAction>();

    toggle() {
      this.drop = !this.drop;
      if (this.drop) {
        this.dropdownActions.emit({action:"dropdown",params:null});
      }
    }
}

It's also happening for collapsibles

I am with an similar issue here. my component is like below:

Drop Me!

one
two

three
view_modulefour
cloudfive

i am using angular4. but if I click on drop_me angular makes a page reload.then if click again on drop me it opens the dropdown structure. how can i fix this ?

I'm having a similar issue with this kind of test :

              menu

              {{b}}

This is a navigationBar that is displayed to the user only when he is successfully logged in
*ngIf="isLoggedIn()"  is supposed to hide it otherwise. But it also bug the dropwdown and I don't see a way to fix it ?

For those trying to do *ngIf="isLoggedIn"
The materealize=dropdown tries to add initiate materialize call before it is conditionally added to the DOM.
you could manually reinitiate the call with materializeAction and javascript timeout in ngOnInit here is an example:
`import { Component, OnInit, EventEmitter} from '@angular/core';
import {Router} from '@angular/router';
import {MaterializeAction} from 'angular2-materialize';
@Component({
selector: 'app-header',
templateUrl: './header.component.html',
styleUrls: ['./header.component.css']
})
export class HeaderComponent implements OnInit {
dropdownActions = new EventEmitter<string|MaterializeAction>();
dropdownParams = [{ inDuration: 300, outDuration: 225, belowOrigin: true}];
constructor(private router:Router) { }
ngOnInit() {
setTimeout(()=>{
this.dropdownActions.emit("dropdown");
}, 20);
}
}
`

I don't know if it is already fixed but my solution was to put 'window.location.reload()' right after the *ngIf is triggered.

The suggested workaround is not false proof.
The following correction is needed to make the workaround work as intended for people using *ngIf="isLoggedIn" or in most other cases as well.
toggle() {
  this.drop = !this.drop;
  if (this.drop) {
    setTimeout(()=>{
      this.dropdownActions.emit({action:"dropdown",params:null});
    });
  }
}

If you flip the boolean value to true and in the next line of code you try to get a reference to the component or DOM element controlled by *ngIf... well, that component or DOM element doesn't exist yet. Angular doesn't run in parallel with your code. Your JavaScript callback has to finish, then Angular (change detection) runs, which will notice the boolean value change and create the component or DOM element and insert it into the DOM.
To fix this issue, call setTimeout(callbackFn, 0) after you flip the boolean value. This adds your callbackFn to the JavaScript message queue. This will ensure that Angular (change detection) runs before your callback function. Hence, when your callbackFn executes, the element on which you want to call materealizeactions should now exist. Using setTimeout(..., 0) ensures that your callbackFn gets called in the next turn of the JavaScript event loop.
Hope this helps.


[Windows] build doesn't output "app-release-signed.apk" - no errors

Describe the bug
Running bubblewrap build --skipPwaValidation appears to successfully complete, but does not
generate a signed apk, only a 'app-release-unsigned-aligned.apk' exists in the directory.
To Reproduce
Steps to reproduce the behavior:

init a new project with no keystore
run bubblewrap build with --skipPwaValidation
check folder for signed apk

Expected behavior
A signed apk named app-release-signed.apk is output or an error is given.
Desktop (please complete the following information):

OS: Windows 10

I'm using --skipPwaValidation as my PWA pass all Installable tests on Chrome Lighthouse.

After futher investigation, it is because find_java.bat does not exist in the android cli tools.
I fixed this problem by hard coding my java .exe in the apksigner.bat file:
rem Check we have a valid Java.exe in the path. set java_exe=C:\Program Files\AdoptOpenJDK\jdk-<IP_ADDRESS>-hotspot\bin\java.exe if exist    "%~dp0..\tools\lib\find_java.bat" call    "%~dp0..\tools\lib\find_java.bat" if exist "%~dp0..\..\tools\lib\find_java.bat" call "%~dp0..\..\tools\lib\find_java.bat" if not defined java_exe goto :EOF
any ideas why find_java.bat does not exist? For the record, I build-tools 29.0.2 and 28.0.3. find_java.bat was missing in both.

This is an issue with the Android command-line tools, so not much we can do in Bubblewrap: https://issuetracker.google.com/issues/150888434
Older versions of the Android SDK tools should work: https://dl.google.com/android/repository/sdk-tools-windows-4333796.zip

A possible workaround is to run apksigner.jar directly, without invoking the .bat:
 java -Xmx1024M -Xss1m -jar "D:\path-to\android-cli-tools\build-tools\29.0.2\lib\apksigner.jar"  sign --ks ./android.keystore --ks-key-alias android --ks-pass pass:android --key-pass pass:android --out ./app-release-signed.apk ./app-release-unsigned-aligned.apk

#191 implements a workaround for this

#191 adds a workaround for this. It's published on v1.3.1. Let me know if this works.


chore: transform libp2p into monorepo

TODO:

[ ] update main README with structure and packages

We may wish to revisit this in future but the repo has moved on quite a bit since this was opened so closing as stale.


bcbio hangs at "summarize fastqc" step

Hi,
We've had a WES variant calling run hang repeatedly at the "summarize fastq" step. We've tried restarting in place a few times, but it just hangs at the same point each time.
I've attached the tail of the traceback from the initial run, and the full traceback of the second attempt (subsequent attempts look much the same).
traceback-tail.txt
traceback2.txt
All the actual results are present in the work directory, but obviously it would be nicer if we could get the run to finish cleanly!
Thanks,
Bennet

Bennet;
Sorry about the issues with hanging. It's tough to tell here what process is taking a long time from just these tracebacks, as they are generic and only tell us we're working in parallel on the QC steps. Would it be possible to run with a single core (-n 1) and then monitor what process (if any) is running when it hangs using top? Then if you exit out with ctrl-c and post the result it should tell us specifically what step bcbio is running. Thanks much for the help debugging.

Closing as this hasn't gotten any action and I think this might be a cluster problem and not something with bcbio. Please re-open if the runs didn't finish or you are still seeing this issue and can provide some more feedback.


Can't build for HIP

What is the expected behavior

That it builds

All output reported here https://github.com/justxi/rocm/issues/132
What actually happens

I get errors

How to reproduce

Try to build

Environment

Hardware
description

GPU
Hawaii PRO [Radeon R9 290/390]

CPU
AMD FX(tm)-8350 Eight-Core Processor

Software
version

ROCK
No idea, kernel 5.6.0-rc1

ROCR
v3.0.0

HCC
v3.0.0

Library
v3.0.0

First, I believe we don't officially support gfx701 anymore.
Second, from the build command, it looks like you are build with hip-clang. Could you let us to know why? Is it possible just build with hcc(remove -DHIP_PLATFORM=hip-clang -DUSE_HIP_CLANG=ON -DHIP_COMPILER=clang)?

Well, given AMD have said that HCC is deprecated and HIP is the future I was trying to build it.
Also, LLVM still supports gfx7 targets.

HCC is not fully deprecated yet. And you should be able to work with it.
Alternatively, if you still want to go for hip-clang,  I would suggest to try at least on gfx803.

Well, I don't have that card, I don't have access to pci atomics either and again, can't upgrade yet, llvm supports gfx7 gcn.

Well, it more looks like a hip-clang build issue. I will have my co-worker to have a look and back to you.

So, this is not a specific issue for rocFFT.
Checked with staff with other teams, "the gfx701 won't be supported by the latest runtime library and hip-clang compiler".
Sorry for the inconvenience.

Firstly it doesn't build no matter what, I already said that.
Surely, just building it with the target set to gfx7 should be ok? The backend supports it.

Not sure about that.
You might post on https://github.com/ROCm-Developer-Tools/HIP.

The build system has changed quite a bit since there was last any activity on this issue.  I'm going to close; please re-open if this is still unresolved.


Update package version to 0.4.2

would just kind of want to make sure that there won't be new security issues raised by this (i assume you did poetry upgrade or something)

Yeah we can iterate if there are other issues that pop up


fix docker run command in README.md

Hi again,
Building inside Docker is super great! I fixed a simple typo of docker run.
before PR

after PR

Cheers!


[beta] Instruction Panel Doesn't Return to Top

Ubuntu 16.04 LTS/Chrome 51
Instructions panel remains scrolled when new challenge is rendered.

fixed in beta


DRILL-5589: JDBC client crashes after successful authentication if tr…

…ace logging is enabled

Thanks! . I added your name in JIRA as reviewer.
Thanks,
Sorabh

From: Karthikeyan Manivannan<EMAIL_ADDRESS>Sent: Thursday, June 15, 2017 7:44:31 PM
To: apache/drill
Cc: Sorabh Hamirwasia; Author
Subject: Re: [apache/drill] DRILL-5589: JDBC client crashes after successful authentication if tr… (#854)
@bitblender commented on this pull request.
LGTM +1
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHubhttps://github.com/apache/drill/pull/854#pullrequestreview-44463788, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AVIgYzw_xX0d3GYuqstvV6viNVYmyjJGks5sEewPgaJpZM4N7r7L.


Add missing value description for modeling nodes

Fixed with https://github.com/spatial-data-lab/knime-geospatial-extension/pull/238


Some refactoring, tests, doc blocks

This refactors the library into separate files per class as suggested by @tm1000 in #18. It also introduces  unit testing with some initial tests. More tests at a deeper level would make sense though (currently it just checks the overall outcome). I also added doc blocks to make things bit easier to understand and for better type hinting in IDEs. This also incorporates #23. After merging, automated unit testing at travis-ci should be set up.
If you're not interested in this, feel free to close.

This is really impressive and should be merged. 👍
One note: Does this require a major version bump? I suppose not. It looks like it's backwards compatible.

Would you mind merging this?

This should be merged. Is this repo managed?

I appreciate this pull request but it is far from the original purpose of Lescript - to be a simple single file without deps.


segfault when run localizer.py

./run.sh: line 58: 13907 Segmentation fault      ./localizer.py -z 32 -e $num_epochs -w $data_dir/macrotrain -tw $data_dir/macrotest -r 0 -s model1.pkl -bgpu -pn 1 -iw $imwidth --serialize 1 ${@:2}

I think it has something to do with GPU

Hard to troubleshoot this without more details on your environment.
Can you try this command from the neon directory and see if it works?
python examples/cifar10_conv.py -e1 -r0 -bgpu

I am using K20, it seems like there is some issue for supporting K20? Is it possible you release some trained model? Or sample results from your model?


Fix Resolv::DNS#timeouts= signature

Motivation
Resolv::DNS#timeouts= also accepts floats:
$ irb
irb(main):001:0> require "resolv"
=> true
irb(main):002:0> dns = Resolv::DNS.new
=> #<Resolv::DNS:0x00007fa3d704bb68 @mutex=#<Thread::Mutex:0x00007fa3d704bb40>, @config=#<Resolv::DNS::Config:0x00007fa3d704bb18 @mutex=#<Thread::Mutex:0x00007fa3d704baf0>, @config_info=nil, @initialized=nil, @timeouts=nil>, @initialized=nil>
irb(main):003:0> dns.timeouts = 0.1
=> 0.1
irb(main):004:0> dns.timeouts = 1
=> 1

Test plan
See included automated tests.

We have a policy of testing changes to Sorbet against Stripe's codebase before
merging them. I've kicked off a test run for the current PR.  When the build
finishes, I'll share with you whether or how it failed. Thanks!
Stripe employees can see the build results here:
→ https://go/builds/bui_JTHwRt85rROAQD
→ https://go/builds/bui_JTHwl7SiRKoadi


categorical market cards look broken in compact view

Vol and event expiration should be at the bottom

these still look broken on local and kovan. I pulled the latest release today

@phoebemirman I'm still seeing this on local. I set it up fresh today but maybe I'm doing something wrong?

closing


Chained "external" autocrafting not working properly - beta-6/7

Issue description:
Chained autocrafting that uses external sources doesn't seem to be working any more.
For instance, to create an improved processor - I have recipes to create the printed silicon, the printed improved processor, and the improved processor itself. If I ask to create an improved processor, it won't automatically request to create the printed silicon or the printed improved processor.
I can ask to create those separately and it'll create them. And if the sub-items already exist, it'll create the improved processor.
What happens:
I request to make an improved processor and it gets "stuck".
What you expected to happen:
I request to make an improved processor and it should request to make each other item in the chain automatically.
Steps to reproduce:

See above description.

...
Version (Make sure you are on the latest version before reporting):

Minecraft: 1.10.2
Forge: 2116
Refined Storage: 1.2-beta7 (and beta6) - It works in beta5.

Does this issue occur on a server? [yes/no] yes
If a (crash)log is relevant for this issue, link it here:
[pastebin/gist/etc link here]

Yes, we just found out about this. Pushing beta 8 now.

Fixed in b175de68a4899a334f7cae765f447b75a2670cdc


chore: more advanced types for sync/async resource

This allows the resource on the span to be sync in the export pipeline while allowing it to be async in the tracer and removes the need for casting.
I only did tracing for the example, but the same should be possible for metrics.

Closed for https://github.com/open-telemetry/opentelemetry-js/pull/1484


C# 9 reference update: lambda discard parameters

Contributes to #20577

@IEvangelist Thank you for review and suggestions.

@BillWagner thank you. I've updated the What's new article with the links (also added the links that should have been added in the previous similar PRs).


add externalLink macro

This PR fix #10 adding print option and externalLink macro.
@pierre-24 : what about the bold font ?

what about the bold font ?

Seems good to me. In any case, it is easy to fix :)


[WIP] Delay VmdbDatabase seed

Depends on https://github.com/ManageIQ/manageiq-ui-classic/pull/2402
This PR is the start of removing VmdbDatabase from the seeding path.  We will still need to do a seed on boot, but I'd like to delay it.  In fact, I'd like to rename "seed" for this class to "refresh" and treat it like inventory.  Since the "inventory" of the database only changes on migrate, it makes sense to do it only once, but I'm not sure when.  I was thinking we would put a refresh on the queue perhaps on evm_server start?  Eventually a generic would pick it up and just do a one-time inventory update.
@jrafanie @kbrock @bdunne  Would like your thoughts here.  And no, we can't just delete it. :smile:

@Fryguy now that ManageIQ/manageiq-ui-classic#2402 has been merged, should this PR have the wip label removed?

No, because we still need to find a way to initially load it, perhaps as a schedule or something...need to discuss with @bdunne and @jrafanie where best to put it for now.

Since the "inventory" of the database only changes on migrate, it makes sense to do it only once, but I'm not sure when

I was under the impression the VmdbDatabase seeding not only seeds metadata but also captures metrics for the tables.  I don't even know the use case.  Don't we show a chart of table size over time if we have the info?  If so, we have to run it on a schedule so we have relevant metrics to plot.  If we don't capture metrics, why is this so slow?

And no, we can't just delete it. 😄

@Fryguy honestly, I don't know what's useful and what's not.  This feature deserves review because I don't know if users even know about it.

test failures look legit.  getting screwed up by lack of seed methon


allow override of cosmwasm commands

Something like:
type Override struct {
	overrideGas uint64
	overrideFee string // coin
	// amount to deposit (for execute contracts)
	// etc.
}

Without this I have to write methods such as ExecuteMsgWithFee, ExecuteMsgWithCoinAmount. But cannot do this with StoreContract easily. Same for InstantiateContract

turns out the newer init function has extraExecTxArgs ...string, we need this for execute msg too