Multiple providers workflow for signed out user

When a signed in user adds a new sign in provider via /auth/:provider/new, they are correctly added to the user_identities table and linked to the logged in user id.
However, consider the following scenario where a logged out user attempts to login using a new authentication provider:

New user registers using Facebook as authentication provider. User is created with email address<EMAIL_ADDRESS>User signs out
User attempts to login using Google as authentication provider which has the same e-mail address as associated with their Facebook account<EMAIL_ADDRESS>Currently, the user is prompted to provide an e-mail address<EMAIL_ADDRESS>on the add_user registration form which then fails with an email "has already been taken" error because there is already a user with an email<EMAIL_ADDRESS>
I want to ensure that a user should only be able to link a new authentication provider when they are already logged in as a user with the same email address. However I would like to be able to let the user know that there is already an account with this e-mail and offer to link them once they authenticate successfully.
Is this possible?

You have two options; build a custom authorization controller or changes how MyApp.UserIdentities.create_user/4 works.
The latter is the easier one as you would only need to override/add MyApp.UserIdentities.create_user/4 method to your user identity context module. You have to be aware that this could potentially open up security flaws. In this case we'll check by using params that comes from the provider. user_id_params are provided by the user in case the user id has already been taken.
def create_user(provider, uid, params, user_id_params) do
  case Users.get_user_by_email(params["email"]) do
    user -> user
    nil -> pow_assent_create_user(provider, uid, params, user_id_params)
  end
end

Creating a custom controller is far more complex as you also have to work around the plug methods. It would give you much better control over security though.
Ultimately, I think that by default requiring users to authenticate before they can add a second authentication to their account is the proper way of handling security.

Oh wait, sorry, I completely misread your question 😄

So I assume you'll show a screen with two options. Sign in to link, or create with new email?
This could be done with a custom controller. Very rough idea of how it could look:
scope "/" do
  pipe_through :browser

  pow_routes()

  scope "/auth", as: "pow_assent" do
    resources "/:provider", PowAssent.Phoenix.AuthorizationController, singleton: true, only: [:new, :delete]
    get "/:provider/callback", PowAssent.Phoenix.AuthorizationController, :callback

    get "/:provider/add-user-id", MyAppWeb.UserIdentityRegistrationController, :add_user_id
    post "/:provider/link", MyAppWeb.UserIdentityRegistrationController, :link
    post "/:provider/create", PowAssent.Phoenix.RegistrationController, :create
  end
end

defmodule MyAppWeb.UserIdentityRegistrationController do
  use MyAppWeb, :controller

  def add_user_id(conn, _params) do
    case user_with_email_exists?(conn.private[:pow_assent_params]) do
       true ->
          changeset = Pow.Plug.change_user(conn, %{"email" => conn.private[:pow_assent_params]["email"]})

          render(conn, "add_user_id.html", changeset: changeset)

      false ->
        Pow.Phoenix.RegistrationController.add_user_id(conn, params)
    end
  end

  def link(conn, %{"provider" => provider, "user" => user_params}) do
    case link_user(conn, provider, user_params)
      {:ok, conn} ->
        PowAssent.Phoenix.AuthorizationController.routes(conn).after_sign_in_path(conn)

      {:error, conn} ->
        render(conn, "add_user_id.html", changeset: Plug.change_user(conn, user_params))
    end
  end

  defp user_with_email_exists?(params), do: # Check if user with email already exists

  defp link_user(conn, user_params) do
    conn
    |> Plug.authenticate_user(user_params)
    |> case do
      {:ok, conn} ->
        uid = conn.private[:pow_assent_params]["uid"]
        user = Pow.Plug.current_user(conn)
        config = Pow.Plug.fetch_config(conn)

        conn
        |> Pow.Plug.current_user()
        |> PowAssent.Operations.create(provider, uid, config)
        |> case do
          {:ok, _user_identity} -> {:ok, user, conn}
          {:error, changeset}   -> {:error, changeset, conn}
        end

      {:error, conn} ->
        {:error, conn}
    end
  end
end

The add_user_id.html.eex template has two options, the original add new user id form, and then one is a sign in form that posts to Routes.pow_assent_registration_path(conn, :link).

That's a great help Dan. I'm trying to replicate the type of workflow that Gravatar uses to link additional authentication providers to an existing account that matches on the e-mail address. See screenshot below.

I was working my way through the custom controllers guide for Pow but this gets me most of the way there. Thanks for the super quick response and code example.

Sounds good! I think this could be good to have as a guide too, so if you're willing to post some sample code when you get it working that would be great.


Repeatable bug.

1: Get over 25 levels.
2: Get three stacks of bottles in your inventory.
3: Only put one stack of bottles on your hotbar, put the rest in your main inventory.

Still needs testing but addressed by #3


Can't compile, glm/vec2 missing

Hi,
I would like to try myself the issues I opened, so I would like to compile it.
Unfotunately, can't because of files missing

hi, sorry i did not see this issue (or the other that you opened!)
did you manage to solve this issue?

no at all, please check

hi, once again, very sorry its taken this long!
the missing glm header was because you needed to install glm using dkp-pacman like so pacman -S switch-glm
if building still isn't working for you, feel free to comment on this issue again and i'll repon it


version number in js

It would be nice to have access to version number from javascript, like ohm.version.
For now I fetch package.json file using ajax:
$.get('https://rawgit.com/harc/ohm/master/package.json', function(json) {
    var term = $('body').terminal(parse, {
        greetings: '[[!;;;;https://github.com/harc/ohm]Ohm Parser] ' + json.version + ' demo'
    });
});

https://codepen.io/jcubic/pen/GMbZvz?editors=0010

Good idea! Done.


Stuck on sync

After ~7 hour of syncing a large Gmail account, no new progress.

CPU still churning.

Had received a Sync error, clicked Try again

Should I let it sit here, worth reviewing logs, or should I restart the app?

@jweinstein gonna merge into #3232
can you post the errors you were having?

It'd be useful to have maybe a progress bar of some description. I'm having a similar issue, but I think it's just taking a long time. If it said the current progress (percent or number of messages?) then it would at least let people know something is happening. My CPU is going, and the icons are flashing... but I still don't know if anything is happening right now. I've got to do this with a few accounts too!


Missing System-Assigned Identity Azure Key Vault Provider for Secrets Store CSI Driver

Hi all,
Based on the resolved issue in secrets-store-csi-driver results in forbidden errors when deployed via Azure ARC onto a K3S cluster the ARM template for the linked page here is missing the identity property to allow a functional deployment:
            "identity": {
                "type": "SystemAssigned"
            },

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: f8b713b9-e735-704a-a0e2-834cfab0260f
Version Independent ID: 3daf6490-1946-b068-bc9f-8fdadfe4aa13
Content: Use Azure Key Vault Secrets Provider extension to fetch secrets into Azure Arc-enabled Kubernetes clusters - Azure Arc
Content Source: articles/azure-arc/kubernetes/tutorial-akv-secrets-provider.md
Service: azure-arc
Sub-service: azure-arc-kubernetes
GitHub Login: @mayurigupta13
Microsoft Alias: mayg

@LeonardHd Thanks for the feedback.
@Jak-MS Thanks for closing it.


Provide the same look-and-feel of Bootstrap popover?

Strongly hope the this great date-time picker can provide the same matching look-and-feel of the official Bootstrap popover. I am using both date-time picker and popover for some inputs in the page, and the styles just do not match.
Currently I am modifying the CSS by myself, but can not get a perfect result, especially the arrow position and size.
Maybe the date-time picker can be wrapped in a popover, instead of a customized widget.

Please add this as a comment to the #841 v5 planning issue. Thanks


Source deploy fails - ERROR running force:source:deploy:  Metadata API request failed:

Summary
Deploying to a sandbox with Salesforce Edge enabled intermittently fails on SFDX but deployment works, causing CI to report incorrectly.
Steps To Reproduce:
Repository to reproduce: dreamhouse-lwc
*NOTE: the deployment does not seem to be relevant, it seems to be the polling logic that can fail when Salesforce Edge returns an unexpected error

Enable Salesfore Edge network on sandbox: Route My Domain Through Salesforce Edge Network.
sfdx force:source:deploy --targetusername DeployUser --sourcepath force-app --testlevel NoTestRun --verbose

Expected result
The deployment status should be returned (either in progress, success, or error) with a retry logic for such network issues on Salesforce.
Actual result
An error is generated, and the process is stopped with the following HTML (the most important part is at the bottom This error originated from Salesforce Edge): sfdxErrorMessage.txt
This seems to be related to #1169 and/or #860.
It was initially logged as support ticket # 42186728.
System Information

It has used purely the sfdx from docker hub: https://hub.docker.com/layers/salesforce/salesforcedx/7.143.0-full/images/sha256-bc22c79683ef61d62b250cc2f9d68b0e24a31f096d43fa5934efc6cf787b1d83

Same here ;) (up)

@renanlq that error got retries added to it a long time ago.  What CLI version are you on?

We are running about: 3060 methods (run local test).
Log:
sfdx-cli/7.186.2 linux-x64 node-v18.13.0
...
sfdx force:source:deploy -p ./build/force-app/main/default -u **** -l RunLocalTests -w 240 --verbose
›   Warning: sfdx-cli update available from 7.186.2 to 7.200.7.
Deploying v56.0 metadata to **** using the v58.0 SOAP API
...
ERROR running force:source:deploy:  Metadata API request failed: FUNCTIONALITY_TEMPORARILY_UNAVAILABLE: Failed while fetching query cursor data for this QueryLocator (org 00DDD000000GjT6, path 0r8xx1i9iU2bq7G), please retry your request.

We are running about: 3060 methods (run local test).
Log:
sfdx-cli/7.186.2 linux-x64 node-v18.13.0
...
sfdx force:source:deploy -p ./build/force-app/main/default -u **** -l RunLocalTests -w 240 --verbose
›   Warning: sfdx-cli update available from 7.186.2 to 7.200.7.
Deploying v56.0 metadata to **** using the v58.0 SOAP API
...
ERROR running force:source:deploy:  Metadata API request failed: FUNCTIONALITY_TEMPORARILY_UNAVAILABLE: Failed while fetching query cursor data for this QueryLocator (org 0*****6, path 0r8xx*****bq7G), please retry your request.

Hi @mshanemc, sorry for, we got some more information and noticed from our site that the problem is about network connection
ERROR running force:source:deploy:  Metadata API request failed: FUNCTIONALITY_TEMPORARILY_UNAVAILABLE: Failed while fetching query cursor data for this QueryLocator (org 00D*******jT6, path 0r8x*******q7G), please retry your request.

So that I'm moving my comments to this topic: https://github.com/forcedotcom/cli/issues/1811, that seems to be similar.

We're still seeing this with<EMAIL_ADDRESS>Almost all our deployment validations fail immediately after starting:
Deploying v57.0 metadata to $USERNAME using the v59.0 SOAP API.
Deploy ID: 0Af8E0000303TvGSAU
Error (1): Metadata API request failed: <!doctypehtml><html lang=en-US><meta charset=UTF-8><title>Error Page</title> ...

The HTML response looks like this when rendered:

Same as @hsmork . Just saw this issue again in VS code today and confirmed the Salesforce Deployment Status page was re-directed to the above "An unexpected connection error occured".
I'm also using<EMAIL_ADDRESS>

Our current workaround is to start validation jobs asynchronously with sf project deploy validate --async and then manually poll for updates with sf project deploy report --job-id "${validation_id}" until the job is done.

Makes sense. Running asynchronously definitely would avoid these types of metadata API request failures that I've seen when running synchronously and the jobs take a bit, especially in production.
But hopefully a new CLI can handle these types of failures running synchronously in the future.


Assess and limit the scope of each token

User Need
As a member of the operations engineering team,
I want to organise and secure the GitHub personal access tokens in the moj-operations-engineering-bot account,
so that we can reduce risks, improve clarity, and align with best practices in token management.
Value
Properly managing these tokens will significantly reduce security risks by limiting their scope and improving their traceability. It will also streamline our processes by creating a clear and efficient method for future token creation and management.
Functional Requirements:

[ ] Assess and limit the scope of each token to minimise potential security risks.
[ ] Scopes updated for least privilege/scopes

Acceptance Criteria:

All token reviewed and set with minimum scopes required for them to work as required.

This is happening as part of individual tickets.


Update the authentication section

Adding detailed information on how to setup app for oauth 2.0 and adding information on windows auth

@mecampos - This is a pretty large change. It also needs to have its images corrected. Can you take this edit?

Sure, I'll work on this PR.
Melissa Campos
she/her
Content Developer 2
Substrate Docs & Community
Office: +506 4114 2639
<EMAIL_ADDRESS>[Microsoft Logo]
From: Jeff Kizner @.>
Sent: Monday, November 8, 2021 12:43 PM
To: MicrosoftDocs/OfficeDocs-MicrosoftSearch @.>
Cc: Melissa Campos (SHE/HER) @.>; Mention @.>
Subject: Re: [MicrosoftDocs/OfficeDocs-MicrosoftSearch] Update the authentication section (PR #207)
@mecamposhttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmecampos&data=04|01|melissa.campos%40microsoft.com|b4cfb5adc02a4994a6ae08d9a2e79557|72f988bf86f141af91ab2d7cd011db47|0|0|637719937817288330|Unknown|TWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D|1000&sdata=83M6v8tLRpPkUXa3KwWM9AN%2FEbriqx7w%2BjOZYHXNaHk%3D&reserved=0 - This is a pretty large change. It also needs to have its images corrected. Can you take this edit?

You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FMicrosoftDocs%2FOfficeDocs-MicrosoftSearch%2Fpull%2F207%23issuecomment-963462466&data=04|01|melissa.campos%40microsoft.com|b4cfb5adc02a4994a6ae08d9a2e79557|72f988bf86f141af91ab2d7cd011db47|0|0|637719937817298324|Unknown|TWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D|1000&sdata=i6henpbDhOX6u2BeKB79EeUiwd7V9rMCuzl83KuKXkE%3D&reserved=0, or unsubscribehttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FASPNO7DDEIYUIZ3LDSES5RDULAK3BANCNFSM5HO2B74A&data=04|01|melissa.campos%40microsoft.com|b4cfb5adc02a4994a6ae08d9a2e79557|72f988bf86f141af91ab2d7cd011db47|0|0|637719937817298324|Unknown|TWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D|1000&sdata=Xp6SuMP%2BZYhzQ3JA5IESkUO6CHLm4ILffxlrTC0zJbw%3D&reserved=0.
Triage notifications on the go with GitHub Mobile for iOShttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fapps.apple.com%2Fapp%2Fapple-store%2Fid1477376905%3Fct%3Dnotification-email%26mt%3D8%26pt%3D524675&data=04|01|melissa.campos%40microsoft.com|b4cfb5adc02a4994a6ae08d9a2e79557|72f988bf86f141af91ab2d7cd011db47|0|0|637719937817308322|Unknown|TWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D|1000&sdata=k1kElcxNkzKwmi0787bDeO1U67WK1v7%2BJHkumtxOqgs%3D&reserved=0 or Androidhttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fplay.google.com%2Fstore%2Fapps%2Fdetails%3Fid%3Dcom.github.android%26referrer%3Dutm_campaign%253Dnotification-email%2526utm_medium%253Demail%2526utm_source%253Dgithub&data=04|01|melissa.campos%40microsoft.com|b4cfb5adc02a4994a6ae08d9a2e79557|72f988bf86f141af91ab2d7cd011db47|0|0|637719937817318320|Unknown|TWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D|1000&sdata=ISwKvT%2BJTWIgqoUrCscanFsZ8a%2BUzONJoeUfWnHRB0Y%3D&reserved=0.

Agreed to submit changes on internal PR#744.

Agreed to submit changes on internal PR#744.

So we can close this PR?

Yes, this PR can be closed now.

Closing per mecampos.


how to feed tensors (with shape of [NCHW]) to the detectors of MMdetection?

Generally,  users are allowed to enter to a path of images to inference_detector(model, img_path), where the path are packaged into a dict of Data_Containers by test_pipeline(XXX) (one Data_Containers is about 'img_metas' and another is 'img').
However, I have to deal with the image tensors firstly with pytorch and  it is quite difficult to feed  image tensors with shape of [N,C,H,W] to detectors directly.  So I wonder how to solve this problem?

In this case, you need to pad the images manually and form them as a batch. We will have better solution in the future


NanLossDuringTrainingError on baseline_lstm_seq2seq w/ and without attention models

I've reported the same error #31closed about 2 weeks ago(and resolved some days after).
Step to reproduce
Launch this script:
pip install -e .

INFO:tensorflow:Creating experiment, storing model files in /root/t2t_train/algorithmic_reverse_decimal40/baseline_lstm_seq2seq-basic_1
INFO:tensorflow:datashard_devices: ['gpu:0']
INFO:tensorflow:caching_devices: None
INFO:tensorflow:Using config: {'_model_dir': '/root/t2t_train/algorithmic_reverse_decimal40/baseline_lstm_seq2seq-basic_1', '_save_checkpoints_secs': 600, '_num_ps_replicas': 0, '_keep_checkpoint_max': 20, '_tf_random_seed': None, '_task_type': None, '_environment': 'local', '_is_chief': True, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7f334a152f90>, '_tf_config': gpu_options {
  per_process_gpu_memory_fraction: 1.0
}
, '_num_worker_replicas': 0, '_task_id': 0, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_evaluation_master': '', '_keep_checkpoint_every_n_hours': 10000, '_master': '', '_session_config': allow_soft_placement: true
graph_options {
  optimizer_options {
  }
}
}
INFO:tensorflow:Performing local training.
WARNING:tensorflow:From /usr/local/lib/python2.7/dist-packages/tensorflow/contrib/learn/python/learn/monitors.py:268: __init__ (from tensorflow.contrib.learn.python.learn.monitors) is deprecated and will be removed after 2016-12-05.
Instructions for updating:
Monitors are deprecated. Please use tf.train.SessionRunHook.
INFO:tensorflow:datashard_devices: ['gpu:0']
INFO:tensorflow:caching_devices: None
INFO:tensorflow:Doing model_fn_body took 0.584 sec.
INFO:tensorflow:This model_fn took 0.907 sec.
INFO:tensorflow:Weight    body/lstm_seq2seq/decoder/rnn/multi_rnn_cell/cell_0/basic_lstm_cell/bias        	shape    (256,)              	size    256
INFO:tensorflow:Weight    body/lstm_seq2seq/decoder/rnn/multi_rnn_cell/cell_0/basic_lstm_cell/kernel      	shape    (128, 256)          	size    32768
INFO:tensorflow:Weight    body/lstm_seq2seq/decoder/rnn/multi_rnn_cell/cell_1/basic_lstm_cell/bias        	shape    (256,)              	size    256
INFO:tensorflow:Weight    body/lstm_seq2seq/decoder/rnn/multi_rnn_cell/cell_1/basic_lstm_cell/kernel      	shape    (128, 256)          	size    32768
INFO:tensorflow:Weight    body/lstm_seq2seq/decoder/rnn/multi_rnn_cell/cell_2/basic_lstm_cell/bias        	shape    (256,)              	size    256
INFO:tensorflow:Weight    body/lstm_seq2seq/decoder/rnn/multi_rnn_cell/cell_2/basic_lstm_cell/kernel      	shape    (128, 256)          	size    32768
INFO:tensorflow:Weight    body/lstm_seq2seq/decoder/rnn/multi_rnn_cell/cell_3/basic_lstm_cell/bias        	shape    (256,)              	size    256
INFO:tensorflow:Weight    body/lstm_seq2seq/decoder/rnn/multi_rnn_cell/cell_3/basic_lstm_cell/kernel      	shape    (128, 256)          	size    32768
INFO:tensorflow:Weight    body/lstm_seq2seq/encoder/rnn/multi_rnn_cell/cell_0/basic_lstm_cell/bias        	shape    (256,)              	size    256
INFO:tensorflow:Weight    body/lstm_seq2seq/encoder/rnn/multi_rnn_cell/cell_0/basic_lstm_cell/kernel      	shape    (128, 256)          	size    32768
INFO:tensorflow:Weight    body/lstm_seq2seq/encoder/rnn/multi_rnn_cell/cell_1/basic_lstm_cell/bias        	shape    (256,)              	size    256
INFO:tensorflow:Weight    body/lstm_seq2seq/encoder/rnn/multi_rnn_cell/cell_1/basic_lstm_cell/kernel      	shape    (128, 256)          	size    32768
INFO:tensorflow:Weight    body/lstm_seq2seq/encoder/rnn/multi_rnn_cell/cell_2/basic_lstm_cell/bias        	shape    (256,)              	size    256
INFO:tensorflow:Weight    body/lstm_seq2seq/encoder/rnn/multi_rnn_cell/cell_2/basic_lstm_cell/kernel      	shape    (128, 256)          	size    32768
INFO:tensorflow:Weight    body/lstm_seq2seq/encoder/rnn/multi_rnn_cell/cell_3/basic_lstm_cell/bias        	shape    (256,)              	size    256
INFO:tensorflow:Weight    body/lstm_seq2seq/encoder/rnn/multi_rnn_cell/cell_3/basic_lstm_cell/kernel      	shape    (128, 256)          	size    32768
INFO:tensorflow:Weight    symbol_modality_12_64/input_emb/weights_0                                       	shape    (1, 64)             	size    64
INFO:tensorflow:Weight    symbol_modality_12_64/input_emb/weights_10                                      	shape    (1, 64)             	size    64
INFO:tensorflow:Weight    symbol_modality_12_64/input_emb/weights_11                                      	shape    (1, 64)             	size    64
INFO:tensorflow:Weight    symbol_modality_12_64/input_emb/weights_12                                      	shape    (0, 64)             	size    0
INFO:tensorflow:Weight    symbol_modality_12_64/input_emb/weights_13                                      	shape    (0, 64)             	size    0
INFO:tensorflow:Weight    symbol_modality_12_64/input_emb/weights_14                                      	shape    (0, 64)             	size    0
INFO:tensorflow:Weight    symbol_modality_12_64/input_emb/weights_15                                      	shape    (0, 64)             	size    0
INFO:tensorflow:Weight    symbol_modality_12_64/input_emb/weights_1                                       	shape    (1, 64)             	size    64
INFO:tensorflow:Weight    symbol_modality_12_64/input_emb/weights_2                                       	shape    (1, 64)             	size    64
INFO:tensorflow:Weight    symbol_modality_12_64/input_emb/weights_3                                       	shape    (1, 64)             	size    64
INFO:tensorflow:Weight    symbol_modality_12_64/input_emb/weights_4                                       	shape    (1, 64)             	size    64
INFO:tensorflow:Weight    symbol_modality_12_64/input_emb/weights_5                                       	shape    (1, 64)             	size    64
INFO:tensorflow:Weight    symbol_modality_12_64/input_emb/weights_6                                       	shape    (1, 64)             	size    64
INFO:tensorflow:Weight    symbol_modality_12_64/input_emb/weights_7                                       	shape    (1, 64)             	size    64
INFO:tensorflow:Weight    symbol_modality_12_64/input_emb/weights_8                                       	shape    (1, 64)             	size    64
INFO:tensorflow:Weight    symbol_modality_12_64/input_emb/weights_9                                       	shape    (1, 64)             	size    64
INFO:tensorflow:Weight    symbol_modality_12_64/softmax/weights_0                                         	shape    (1, 64)             	size    64
INFO:tensorflow:Weight    symbol_modality_12_64/softmax/weights_10                                        	shape    (1, 64)             	size    64
INFO:tensorflow:Weight    symbol_modality_12_64/softmax/weights_11                                        	shape    (1, 64)             	size    64
INFO:tensorflow:Weight    symbol_modality_12_64/softmax/weights_12                                        	shape    (0, 64)             	size    0
INFO:tensorflow:Weight    symbol_modality_12_64/softmax/weights_13                                        	shape    (0, 64)             	size    0
INFO:tensorflow:Weight    symbol_modality_12_64/softmax/weights_14                                        	shape    (0, 64)             	size    0
INFO:tensorflow:Weight    symbol_modality_12_64/softmax/weights_15                                        	shape    (0, 64)             	size    0
INFO:tensorflow:Weight    symbol_modality_12_64/softmax/weights_1                                         	shape    (1, 64)             	size    64
INFO:tensorflow:Weight    symbol_modality_12_64/softmax/weights_2                                         	shape    (1, 64)             	size    64
INFO:tensorflow:Weight    symbol_modality_12_64/softmax/weights_3                                         	shape    (1, 64)             	size    64
INFO:tensorflow:Weight    symbol_modality_12_64/softmax/weights_4                                         	shape    (1, 64)             	size    64
INFO:tensorflow:Weight    symbol_modality_12_64/softmax/weights_5                                         	shape    (1, 64)             	size    64
INFO:tensorflow:Weight    symbol_modality_12_64/softmax/weights_6                                         	shape    (1, 64)             	size    64
INFO:tensorflow:Weight    symbol_modality_12_64/softmax/weights_7                                         	shape    (1, 64)             	size    64
INFO:tensorflow:Weight    symbol_modality_12_64/softmax/weights_8                                         	shape    (1, 64)             	size    64
INFO:tensorflow:Weight    symbol_modality_12_64/softmax/weights_9                                         	shape    (1, 64)             	size    64
INFO:tensorflow:Weight    symbol_modality_12_64/target_emb/weights_0                                      	shape    (1, 64)             	size    64
INFO:tensorflow:Weight    symbol_modality_12_64/target_emb/weights_10                                     	shape    (1, 64)             	size    64
INFO:tensorflow:Weight    symbol_modality_12_64/target_emb/weights_11                                     	shape    (1, 64)             	size    64
INFO:tensorflow:Weight    symbol_modality_12_64/target_emb/weights_12                                     	shape    (0, 64)             	size    0
INFO:tensorflow:Weight    symbol_modality_12_64/target_emb/weights_13                                     	shape    (0, 64)             	size    0
INFO:tensorflow:Weight    symbol_modality_12_64/target_emb/weights_14                                     	shape    (0, 64)             	size    0
INFO:tensorflow:Weight    symbol_modality_12_64/target_emb/weights_15                                     	shape    (0, 64)             	size    0
INFO:tensorflow:Weight    symbol_modality_12_64/target_emb/weights_1                                      	shape    (1, 64)             	size    64
INFO:tensorflow:Weight    symbol_modality_12_64/target_emb/weights_2                                      	shape    (1, 64)             	size    64
INFO:tensorflow:Weight    symbol_modality_12_64/target_emb/weights_3                                      	shape    (1, 64)             	size    64
INFO:tensorflow:Weight    symbol_modality_12_64/target_emb/weights_4                                      	shape    (1, 64)             	size    64
INFO:tensorflow:Weight    symbol_modality_12_64/target_emb/weights_5                                      	shape    (1, 64)             	size    64
INFO:tensorflow:Weight    symbol_modality_12_64/target_emb/weights_6                                      	shape    (1, 64)             	size    64
INFO:tensorflow:Weight    symbol_modality_12_64/target_emb/weights_7                                      	shape    (1, 64)             	size    64
INFO:tensorflow:Weight    symbol_modality_12_64/target_emb/weights_8                                      	shape    (1, 64)             	size    64
INFO:tensorflow:Weight    symbol_modality_12_64/target_emb/weights_9                                      	shape    (1, 64)             	size    64
INFO:tensorflow:Total trainable variables size: 266496
INFO:tensorflow:Computing gradients for global model_fn.
INFO:tensorflow:Global model_fn finished.
INFO:tensorflow:Create CheckpointSaverHook.
2017-07-07 09:57:25.953248: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.
2017-07-07 09:57:25.953294: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.
2017-07-07 09:57:25.953312: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.
2017-07-07 09:57:25.988581: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:893] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2017-07-07 09:57:25.988864: I tensorflow/core/common_runtime/gpu/gpu_device.cc:940] Found device 0 with properties:
name: GeForce GTX 670MX
major: 3 minor: 0 memoryClockRate (GHz) 0.601
pciBusID 0000:01:00.0
Total memory: 2.94GiB
Free memory: 2.62GiB
2017-07-07 09:57:25.988902: I tensorflow/core/common_runtime/gpu/gpu_device.cc:961] DMA: 0
2017-07-07 09:57:25.988916: I tensorflow/core/common_runtime/gpu/gpu_device.cc:971] 0:   Y
2017-07-07 09:57:25.988936: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1030] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX 670MX, pci bus id: 0000:01:00.0)
2017-07-07 09:57:30.101101: I tensorflow/core/common_runtime/gpu/pool_allocator.cc:247] PoolAllocator: After 5206 get requests, put_count=3349 evicted_count=1000 eviction_rate=0.298597 and unsatisfied allocation rate=0.567998
2017-07-07 09:57:30.101172: I tensorflow/core/common_runtime/gpu/pool_allocator.cc:259] Raising pool_size_limit_ from 100 to 110
INFO:tensorflow:Saving checkpoints for 1 into /root/t2t_train/algorithmic_reverse_decimal40/baseline_lstm_seq2seq-basic_1/model.ckpt.
ERROR:tensorflow:Model diverged with loss = NaN.
Traceback (most recent call last):
  File "/usr/local/bin/t2t-trainer", line 6, in <module>
    exec(compile(open(__file__).read(), __file__, 'exec'))
  File "/t2t/tensor2tensor/bin/t2t-trainer", line 83, in <module>
    tf.app.run()
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/platform/app.py", line 48, in run
    _sys.exit(main(_sys.argv[:1] + flags_passthrough))
  File "/t2t/tensor2tensor/bin/t2t-trainer", line 79, in main
    schedule=FLAGS.schedule)
  File "/t2t/tensor2tensor/utils/trainer_utils.py", line 246, in run
    run_locally(exp_fn(output_dir))
  File "/t2t/tensor2tensor/utils/trainer_utils.py", line 536, in run_locally
    exp.train_and_evaluate()
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/learn/python/learn/experiment.py", line 495, in train_and_evaluate
    self.train(delay_secs=0)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/learn/python/learn/experiment.py", line 275, in train
    hooks=self._train_monitors + extra_hooks)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/learn/python/learn/experiment.py", line 665, in _call_train
    monitors=hooks)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/util/deprecation.py", line 289, in new_func
    return func(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/learn/python/learn/estimators/estimator.py", line 455, in fit
    loss = self._train_model(input_fn=input_fn, hooks=hooks)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/learn/python/learn/estimators/estimator.py", line 1007, in _train_model
    _, loss = mon_sess.run([model_fn_ops.train_op, model_fn_ops.loss])
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/monitored_session.py", line 505, in run
    run_metadata=run_metadata)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/monitored_session.py", line 842, in run
    run_metadata=run_metadata)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/monitored_session.py", line 798, in run
    return self._sess.run(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/monitored_session.py", line 960, in run
    run_metadata=run_metadata))
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/basic_session_run_hooks.py", line 477, in after_run
    raise NanLossDuringTrainingError
tensorflow.python.training.basic_session_run_hooks.NanLossDuringTrainingError: NaN loss during training.

After changing the problem to algorithmic_reverse_nlplike_decimal8K finally runs. So probably the error is tied to the weights initialization(saturating the gradient), a bad choice of the hparams or some NaN input. I will check it out.

Yes the problem was with weight_decay being >1 which when combined with L2 loss caused the total loss to blow up.
In the latest version it is set to 0.  I'm not getting NanLossDuringTrainingError anymore. Can you confirm?


Allow the dashboard header to be set manually

Description
Allow the dashboard header to be set manually instead of using the Controller name
Previously, the controller name was used for the dashboard page header. However if the controller was more than one word long then the header would be the Controller name without any spaces

Wouldn't it make sense then to also Inflectlor::humanize() this default value?

[Current coverage][cc-pull] is 19.56%

Merging [#150][cc-pull] into [master][cc-base-branch] will decrease coverage by 76.94%

@@           master       #150   diff @@
========================================
  Files          12         12
  Lines         315        409    +94
  Methods        37         37
  Branches        0          0
========================================
- Hits          304         80   -224
- Misses         11        329   +318
  Partials        0          0

Powered by Codecov. Last updated by [d148323...9359c25][cc-compare]
[cc-base-branch]: https://codecov.io/gh/FriendsOfCake/bootstrap-ui/branch/master?src=pr
[cc-compare]: https://codecov.io/gh/FriendsOfCake/bootstrap-ui/compare/d1483230d1bb46f6ff114ae987ddefd285f09dc7...9359c25f693d4fb1f6c3ecc080c6a95d636657a1
[cc-pull]: https://codecov.io/gh/FriendsOfCake/bootstrap-ui/pull/150?src=pr

@dereuromark humanise will replace underscored words, in my case the controller name appears in camel case. Do you know any methods to easily separate that?

Humanize does separate via spaces :) See the test cases - or http://sandbox.dereuromark.de/sandbox/inflector?string=hello_world

$this->request->controller gives 'HelloWorld'. So it would take 2 lines - underscoring and then humanising. Is that what you mean?

Was just an idea to resolve the multi word issue as a default value :)

Yes, it's a good idea! I'll make the change

Any update?

Closing due to lack of follow up.


[feature] fix code to comply PSR-2, add code style check to travis

Codecov Report

Merging #31 into master will not change coverage.
The diff coverage is n/a.

@@            Coverage Diff            @@
##             master      #31   +/-   ##
=========================================
  Coverage     95.17%   95.17%
  Complexity      877      877
=========================================
  Files            81       81
  Lines          2300     2300
=========================================
  Hits           2189     2189
  Misses          111      111

Impacted Files
Coverage Δ
Complexity Δ

src/Command/Branch/ContextSequence.php
85.29% <ø> (ø)
18 <0> (ø)
:arrow_down:

src/Mapper/Mapper.php
100% <ø> (ø)
11 <0> (ø)
:arrow_down:

src/Parser/Typecast.php
91.3% <ø> (ø)
14 <0> (ø)
:arrow_down:

src/Select/Loader/HasManyLoader.php
96.77% <ø> (ø)
7 <0> (ø)
:arrow_down:

src/Heap/Traits/VisitorTrait.php
100% <ø> (ø)
2 <0> (ø)
:arrow_down:

src/Command/Branch/Condition.php
100% <ø> (ø)
4 <0> (ø)
:arrow_down:

src/Relation/Morphed/MorphedHasMany.php
95.45% <ø> (ø)
6 <0> (ø)
:arrow_down:

src/Mapper/DatabaseMapper.php
98.57% <ø> (ø)
17 <0> (ø)
:arrow_down:

src/Select/Traits/WhereTrait.php
100% <ø> (ø)
2 <0> (ø)
:arrow_down:

src/Command/Branch/Split.php
100% <ø> (ø)
10 <0> (ø)
:arrow_down:

... and 63 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d493e8a...133bf81. Read the comment docs.


clientconfig.clear_cache does not clear the cache when ClientConfig settings are saved

Summary
ClientConfig's system setting clientconfig.clear_cache does not clear the cache when ClientConfig settings are saved
Step to reproduce
Set clientconfig.clear_cache to Yes [default]
Observed behavior
When saving a ClientConfig setting, the cache should be cleared [as I understand it] - but in all my tests, it's still necessary to manually clear the cache or call the setting uncached.
Expected behavior
MODX cache should be cleared
Environment
ClientConfig version 2.4.0-pl,
MODX version 3.0.4-pl,
PHP version 8.2.

Confirmed. Bumping it :)

Just to confirm, you're saying it does not clear the resource cache (or a different one), and this is all with the standard cache setups?

Just to confirm, you're saying it does not clear the resource cache (or a different one), and this is all with the standard cache setups?

I can just tell for myself, but I am running a default MODX 3.0.4. clientconfig.clear_cache is set to Yes. I render an example client config setting in the default template. When I change the value in client config and save it I would expect the cache to be cleared instantly. Instead the old values are still visible until I clear the cache either manually, or save a resource/chunk/snippet.

That's the experience I have as well. Standard cache setup.

There was some discussion here

There was some discussion here

Yes, this is clearly a bug...I would expect client config to clear the whole cache after save when this setting is enabled.

Running into this issue too. Possibly having something to do with
https://github.com/modmore/ClientConfig/blob/master/core/components/clientconfig/processors/mgr/settings/save.class.php#L73
From the modx error log:
PHP warning: rmdir(/.../core/cache/resource/web/resources/): Directory not empty
PHP warning: rmdir(/.../core/cache/resource/web/): Directory not empty

Current workaround is to create a plugin on the ClientConfig_ConfigChange event and manually clearing the resource folder.

Correct @verstratenbram - that's exactly the line that is supposed to wipe the resource cache clean and I'm not sure why it wouldn't. 😂
I've wondered if we need to change that to use the refresh() instead:
$this->modx->getCacheManager()->refresh([
    'resource' => [],
]);

Hi @Mark-H, that seems to resolve the issue.
Don't understand why the original code doesn't work though.

I've looked a bit further into this issue.
The previous error messages Directory not empty can be ignored. This was because we are using a custom resource cache handler which uses the .cache.gz extension for cache files instead of the default .cache.php. Because of that the files were not getting deleted for some modCacheManager functions. We have resolved this issue by using the default extension instead.
There is still something odd happening though. Upon saving the client configuration and immediately reloading the front-end, the old values persist. Yet, when waiting for 2-3 seconds after saving and then reloading the front-end, the correct values appear.
Just to be clear, the resource cache does get cleared, but it gets filled with the old client config values if you refresh the front-end page fast enough.


If you click refresh while a stash is expanded, you can't see the files

Initial state:

After clicking refresh:

They don't go back to normal until you collapse them, and refresh again.

Sorry for late replying. This got fixed on v4.0.0. Feel free to open a new issue if you encounter any problem


Building x86_64 on M1 fails.

Attempting to build an x86_64 binary with
./waf configure --disable-debug-build --swift-flags="-target x86_64-apple-macos10.12"
throws an error during linking

[225/225] Linking build/mpv
ld: warning: ignoring file osdep/macOS_swift.o, building for macOS-arm64 but attempting to link with file built for unknown-x86_64
Undefined symbols for architecture arm64:
"OBJC_CLASS$__TtC11macOS_swift19RemoteCommandCenter", referenced from:
objc-class-ref in macosx_events.m.22.o
"OBJC_CLASS$__TtC11macOS_swift7CocoaCB", referenced from:
objc-class-ref in macosx_application.m.22.o
ld: symbol(s) not found for architecture arm64
clang: error: linker command failed with exit code 1 (use -v to see invocation)

Compiling for ARM works fine. I'm not an expert so this could simply be a case of a missing flag, but I'm stumped.

I'm sorry, I posted this in the wrong repo, closing!


Change repo to be private when GHE is private mode.

When the case of sending pull-request to repository on private-mode Github:enterprise,
build of origin fails because it can't checkount repository.
(build of forked repository will success)
It is because of builder writes private-key when repository is private,
however, private-mode ghe's repository is not flagged as private,
so builder doesn't write private-key.

yes, great catch. thanks!


tf.estimator.train_and_evaluate got wrong Evaluation accuray and loss

I use tf.estimator.train_and_evaluate to train and evaluate my model. This is my code:
def model_fn(features, labels, mode):
    is_training = (mode == tf.estimator.ModeKeys.TRAIN)
    with slim.arg_scope(resnet_v2.resnet_arg_scope()):
        logits, endpoints = resnet_v2.resnet_v2_50(features, num_classes=cls_num,
                is_training=is_training)
    logits = tf.squeeze(logits, [1, 2])
    preds = tf.argmax(logits, 1)
    loss = tf.losses.sparse_softmax_cross_entropy(labels=labels, logits=logits)
    accuracy = tf.metrics.accuracy(labels=labels, predictions=preds)
    metrics = {'accuracy': accuracy}
    if mode == tf.estimator.ModeKeys.EVAL:
        return tf.estimator.EstimatorSpec(mode, loss=loss, eval_metric_ops=metrics)
    optimizer = tf.train.AdagradOptimizer(learning_rate=0.1)
    train_op = optimizer.minimize(loss, global_step=tf.train.get_global_step())
    return tf.estimator.EstimatorSpec(mode, loss=loss, train_op=train_op)

def train_input_fn():
    dataset = tf.data.Dataset.from_tensor_slices((np.array(img_train), label_train))
    dataset = dataset.repeat(5).batch(8)
    return dataset

def eval_input_fn():
    dataset = tf.data.Dataset.from_tensor_slices((np.array(img_test), label_test))
    dataset = dataset.repeat(1).batch(8)
    return dataset

estimator = tf.estimator.Estimator(model_fn=model_fn, model_dir='logs')
train_spec = tf.estimator.TrainSpec(input_fn=train_input_fn)
eval_specs = tf.estimator.EvalSpec(input_fn=eval_input_fn)
tf.estimator.train_and_evaluate(estimator, train_spec, eval_specs)

The training step is ok and the loss became very small (about 0.001), but the evaluation result is wrong (the following is the evaluaiton log):
...
INFO:tensorflow:Saving dict for global step 625: accuracy = 0.5, global_step = 625, loss =<PHONE_NUMBER>000.0
...

The task is very simple, just a binaray classfication. I don not think it is overfitting. Is there something wrong for my evaluation code?

@aravind3134 decreasing the learning rate to 0.001 seems not work:
INFO:tensorflow:Saving dict for global step 625: accuracy = 0.5, global_step = 625, loss = 7749.7485

The loss is smaller but the accuracy is also 0.5.

Can you let me know the data-set size and can I see more logs in the console, if you have any.
You have them probably above this line:
INFO:tensorflow:Saving dict for global step 625: accuracy = 0.5, global_step = 625, loss = 7749.7485
If you don't find any, you can edit line
train_spec = tf.estimator.TrainSpec(input_fn=train_input_fn)   to
logging_hook = tf.train.LoggingTensorHook (tensors=tensors_to_log, every_n_iter=50)
train_spec = tf.estimator.TrainSpec(input_fn=train_input_fn, max_steps=1000, hook = [logging_hook])
This should give you the summary of the training you are doing.
See if your loss during your training is fluctuating. If it is, you might have to look for a better learning rate, may be by performing a grid search on the interval [0.00001, 0.001].
The loss seems to have decreased a lot, but nothing can be judged with one line. Let me know more about the logs.
Hope this helps.

OK, this is my training log:
INFO:tensorflow:Using default config.
INFO:tensorflow:Using config: {'_save_checkpoints_secs': 600, '_session_config': None, '_keep_checkpoint_max': 5, '_task_type': 'worker', '_global_id_in_cluster': 0, '_is_chief': True, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7fefaadf1ed0>, '_evaluation_master': '', '_save_checkpoints_steps': None, '_keep_checkpoint_every_n_hours': 10000, '_service': None, '_num_ps_replicas': 0, '_tf_random_seed': None, '_master': '', '_device_fn': None, '_num_worker_replicas': 1, '_task_id': 0, '_log_step_count_steps': 100, '_model_dir': 'logs', '_train_distribute': None, '_save_summary_steps': 100}
INFO:tensorflow:Running training and evaluation locally (non-distributed).
INFO:tensorflow:Start train and evaluate loop. The evaluate will happen after every checkpoint. Checkpoint frequency is determined based on RunConfig arguments: save_checkpoints_steps None or save_checkpoints_secs 600.
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Create CheckpointSaverHook.
INFO:tensorflow:Graph was finalized.
2018-08-16 09:47:59.546073: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2018-08-16 09:48:01.441117: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1405] Found device 0 with properties:
name: Tesla P40 major: 6 minor: 1 memoryClockRate(GHz): 1.531
pciBusID: 0000:83:00.0
totalMemory: 22.38GiB freeMemory: 22.21GiB
2018-08-16 09:48:01.441186: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1484] Adding visible gpu devices: 0
2018-08-16 09:48:01.811401: I tensorflow/core/common_runtime/gpu/gpu_device.cc:965] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-08-16 09:48:01.811463: I tensorflow/core/common_runtime/gpu/gpu_device.cc:971]      0
2018-08-16 09:48:01.811477: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] 0:   N
2018-08-16 09:48:01.812022: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1097] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 21544 MB memory) -> physical GPU (device: 0, name: Tesla P40, pci bus id: 0000:83:00.0, compute capability: 6.1)
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Saving checkpoints for 0 into logs/model.ckpt.
INFO:tensorflow:loss = 0.7005657, step = 0
INFO:tensorflow:global_step/sec: 13.5045
INFO:tensorflow:loss = 0.3813903, step = 100 (7.405 sec)
INFO:tensorflow:global_step/sec: 15.0218
INFO:tensorflow:loss = 0.5762321, step = 200 (6.657 sec)
INFO:tensorflow:global_step/sec: 15.0521
INFO:tensorflow:loss = 0.15642221, step = 300 (6.644 sec)
INFO:tensorflow:global_step/sec: 15.0839
INFO:tensorflow:loss = 0.19744259, step = 400 (6.630 sec)
INFO:tensorflow:global_step/sec: 15.0664
INFO:tensorflow:loss = 0.110218376, step = 500 (6.638 sec)
INFO:tensorflow:global_step/sec: 15.0758
INFO:tensorflow:loss = 0.04312685, step = 600 (6.633 sec)
INFO:tensorflow:Saving checkpoints for 625 into logs/model.ckpt.
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Starting evaluation at 2018-08-16-01:50:14
INFO:tensorflow:Graph was finalized.
2018-08-16 09:50:15.665224: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1484] Adding visible gpu devices: 0
2018-08-16 09:50:15.665298: I tensorflow/core/common_runtime/gpu/gpu_device.cc:965] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-08-16 09:50:15.665314: I tensorflow/core/common_runtime/gpu/gpu_device.cc:971]      0
2018-08-16 09:50:15.665324: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] 0:   N
2018-08-16 09:50:15.665545: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1097] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 21544 MB memory) -> physical GPU (device: 0, name: Tesla P40, pci bus id: 0000:83:00.0, compute capability: 6.1)
INFO:tensorflow:Restoring parameters from logs/model.ckpt-625
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Evaluation [10/100]
INFO:tensorflow:Evaluation [20/100]
INFO:tensorflow:Evaluation [30/100]
INFO:tensorflow:Evaluation [40/100]
INFO:tensorflow:Evaluation [50/100]
INFO:tensorflow:Evaluation [60/100]
INFO:tensorflow:Evaluation [70/100]
INFO:tensorflow:Finished evaluation at 2018-08-16-01:50:19
INFO:tensorflow:Saving dict for global step 625: accuracy = 0.5, global_step = 625, loss = 7912.4326
INFO:tensorflow:Saving 'checkpoint_path' summary for global step 625: logs/model.ckpt-625
INFO:tensorflow:Loss for final step: 0.028676154.

...

One thing to look out for is making sure the shapes of the predictions and labels being fed into the accuracy metric are the same. There could be some broadcasting at times that can lead to odd values for the accuracy number. Use tf.shape. tf.Print to print out for a few steps and see whats going on.

@honeytidy, did you resolve this issue? I am having a very similar issue. In fact, I replaced the eval dataset with the training dataset and I still get different training and eval losses on the same dataset.

@honeytidy did you solve your problem?

I experience a similar problem where the evaluation loss reported during training (using estimator.train_and_evaluate())at a given step is much higher than when I evaluate the checkpoint at this step using estimator.evaluate(). The datasets and EvalSpec are exactly the same. I already changed the steps paramter in the EvalSpec to None. Is this behavior expected?

I had a similar problem and it turned out to be related to the batch normalization module in my code that the fact that I needed to update the moving average and variance each time before training. basically adding the following commands would fix this issue.
   optimizer = tf.train.AdamOptimizer(FLAGS.learning_rate)
    with tf.control_dependencies(tf.get_collection(tf.GraphKeys.UPDATE_OPS)):
        train_op = optimizer.minimize(loss_fn, var_list=tf.trainable_variables())


Make sure there is an ingressclass for Traefik

otherwise the Ingresses created by cert-manager for the challenges,
won't work. That's because they get the new "ingressClass" field which
requires an Ingress Class to exist.
More here:

https://github.com/traefik/traefik/blob/4235cef1b239bdffdb2fe7c62ebcac83df55781f/docs/content/providers/kubernetes-ingress.md#ingressclass
https://github.com/traefik/traefik-helm-chart/blob/master/traefik/values.yaml#L64
https://kubernetes.io/blog/2020/04/02/improvements-to-the-ingress-api-in-kubernetes-1.18/#specifying-the-class-of-an-ingress
https://cert-manager.io/docs/usage/ingress/

Probably cert-manager started adding that field here:
https://github.com/jetstack/cert-manager/commit/57e9e57fbf015783da258e40fe8f932f89f4fdac

Just tested on GKE and it works perfectly!

Deploying the sample-app worked too.


Added tests for tensors, inputs, outputs, opcodes

This was one of the TODO, just complete the same

@alanchiao Could you PTAL and approve.

@alanchiao can you please review this PR ?

Nagging Reviewer @alanchiao: You have been added as a reviewer to this pull request. Please add your review or reassign. It has been 15 days with no activity and the awaiting review label has been applied.

Nagging Reviewer @alanchiao: You have been added as a reviewer to this pull request. Please add your review or reassign. It has been 30 days with no activity and the awaiting review label has been applied.

Can one of the admins verify this patch?

Closing as a duplicate of https://github.com/tensorflow/tensorflow/pull/25809


init][ 328]: AX_ENGINE_CreateHandle [E][Init][ 124]: init axmodel(tinyllama-int8/tinyllama_l1.axmodel) failed

Running it once or twice and then running it again would cause such errors, requiring a restart of the board each time it was encountered.

if you used AX650N, please check it bsp version is >= v1.45.0_P38
using the llm build guide to get the new axmodel of TinyLlama1.1


Example: How to create and register a subliminal provider extension.

If you are like me, you want to make your own provider but you don't want to fork or wrap subliminal.
For anyone wondering how to make their own provider 'extension', you may have noticed that documentation for this is a placeholder:
https://subliminal.readthedocs.io/en/latest/api/extensions.html
**Extensions
Extension managers for the providers, refiners and language converters.

But don't fret, after many trials and tribulations I have figured out how to perform the magical incantations necessary to make an extension.
All you have to do is make an entry in your pyproject.toml like so:
[project.entry-points."subliminal.providers"]
dummy = "ihatestevedore.dummyprovider:DummyProvider"

In this example, your package name is ihatestevedore and it has a module named dummyprovider containing your provider class definition. The entry name that you assign (left side of the '=') is arbitrary and can be whatever you want. You can then create your package layout like so:
/path/to/extension/ihatestevedore
└───ihatestevedore
    │   dummyprovider.py

Where you implement the DummyProvider class in the dummyprovider module:
from subliminal import Provider

class DummyProvider(Provider):
    pass

Here is what a complete project.toml might look like:
[project]
name = "ihatestevedore"
version = "0.1.0"
description = "Add your description here"
readme = "README.md"
requires-python = ">=3.12"
dependencies = [
    "subliminal>=2.2.1",
]

[project.entry-points."subliminal.providers"]
dummy = "ihatestevedore.dummyprovider:DummyProvider"

[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

After installing your package, there should be an entry_points.txt within its home in the site-packages directory:
$ ls python/Lib/site-packages/ihatestevedore-0.1.0.dist-info
direct_url.json
entry_points.txt
INSTALLER
METADATA
RECORD
REQUESTED
WHEEL

It should look like:
[subliminal.providers]
dummy = ihatestevedore.dummyprovider:DummyProvider

Now your extension should be available for discovery by subliminal and you can follow the rest of the docs and examples concerning making your own provider.
More reading on how this works:

making plugins with stevedore https://docs.openstack.org/stevedore/latest/user/tutorial/creating_plugins.html
python toml entrypoints spec https://packaging.python.org/en/latest/specifications/entry-points/#entry-points
subliminal's wrapper of stevedore https://github.com/Diaoul/subliminal/blob/2.2.1/subliminal/extensions.py

Thanks for bringing this up :)
About the documentation, the extensions section has more information but it is not updated on Readthedocs (@Diaoul).
But there is a section about how to write your own provider here:
https://subliminal.readthedocs.io/en/latest/user/provider_guide.html
Thish doc section explains how to write a provider but not what to do after:

make a PR with the new provider to make it available from subliminal (it can be a long process).
write your own python package for the provider with an entry point to be picked by subliminal.
(a third option, that doesn't exist yet, would be to make a PR to a python package (to be created) that has several providers but with an easier and faster merge process.)

Anyway, it would be great to add the information you give to the docs actually (if you have time for a PR :))

I'd be willing to submit a PR once I have my provider working.
My own take on this is that adding information to the docs about extensions might be a touch premature. The cli module currently has no support for passing arguments to a provider.
Getting args passed from the cli to my provider required some hackery:
import click

from click import get_current_context
from subliminal.cli import (
    providers_config,
)
from subliminal import cli

# pull out the function wrapped by the command and other decorators.
# Remember to unreverse the params before reattaching them to the
# replacement command.

# Fore sure breaking the original command group in some way here.
# There might be a way to reuse the existing group.
@click.group(
    name=cli.subliminal.name,
    params=list(reversed(cli.subliminal.params)),
    context_settings=cli.subliminal.context_settings,
    epilog=cli.subliminal.epilog,
)
# this ends up creating a whole new command group in the generated docs
# for some reason
@providers_config.option(
    '--jimaku',
    type=click.STRING,
    nargs=2,
    metavar='JIMAKU_APIKEY',
    help='Jimaku API key.',
)
def root_cmd_wraper(*args, **kwargs):
    restore_kwargs = {**kwargs}
    jimaku_args = restore_kwargs.pop('jimaku', None)

    if jimaku_args:
        # setup provider config here
        pass

    ctx = get_current_context()
    cli.subliminal.callback.__wrapped__(
        ctx, *args, **restore_kwargs
    )

# monkey patch subliminal
cli.subliminal = root_cmd_wraper

Do we really want to add something like: "to pass args to your provider, reverse engineer click and monkey patch core" to the official docs?
The extensions part of the docs being a stub is a stern, albeit indirect, indication that they're not fully developed to anyone thinking of writing one. Were there to be a guide on how to write an extension and then omit how to pass args with the cli, that feels like I'd be dead ending any hopeful developers.
Maybe a new section of the docs like 'experimental', 'only if you dare' or a clear demarcation at the top of the extensions section would help?
I hate to be clobbering github issues with non tickets like this. For the time being, this is the only place we have to collaborate on stuff right now.  Would you mind keeping this one open?

You cannot pass arguments to the providers directly from the CLI (I am still thinking what would be the best way to do it) but you can pass them through the configuration file:
https://github.com/Diaoul/subliminal/blob/main/docs/config.toml
Create a section [provider.my_provider] and add the keyword arguments you need.
The documentation lacking clarity is enough reason to open an issue ;)
We also opened a Discord channel:
https://discord.com/invite/kXW6sWte9N

I've collected all the hacks I did to get my provider to actually run, check this link  for details.
There is one additional problem that is not addressed by #1180 making it impossible for providers from separate packages to work. Even though provider entry points are discovered by the extension manager, they are not passed to the AsyncProviderPool.
Extensions have to be 'registered' yet again using by calling the register() method on the extension manager before they are actually used.
I monkey patched click.option() to force the registration. Focus on this part:
provider_manager.register('jimaku = sublimaku:JimakuProvider')

Within this code:
_option = click.option

def option(*args, **kwargs):
    global provider_group
    # force the download subcommand to use make the jimaku provider available
    if '--provider' in args:
        provider_manager = globals().get('provider_manager')
        if provider_manager is None:
            from subliminal import provider_manager
        provider_manager.register('jimaku = sublimaku:JimakuProvider')
        kwargs['type'] = click.Choice(sorted(provider_manager.names()))

    # save a copy of the providers_config group so we can add our own
    # options to it later
    if provider_group is None and (group := kwargs.get('group')):
        if isinstance(group, OptionGroup) and group.name == 'Providers configuration':
            provider_group = group

    return _option(*args, **kwargs)

click.option = option

I'd argue that requiring a call on extension_manager.register() isn't adding any value, nor is it adding protection, as evidenced by the above code. Keeping core extensions separate from others would certainly come to have some merit in the future, but going through all the work of adding cross package discoverability and then switching it off seems a little bit odd.

I just tried creating a simple package with this entry-point in its pyproject.toml:
[project.entry-points."subliminal.providers"]
mytestprovider = "mytestprovider.provider:MyTestProviderProvider"

Then I install the package (in the same virtualenv as subliminal) and it is recognized by subliminal, it appears in the list of authorized providers (also with #1180, the arguments of mytestproviders are listed in the help):
$ subliminal download --help
...
  -p, --provider [ALL|addic7ed|bsplayer|gestdown|mytestprovider|napiprojekt|opensubtitles|opensubtitlescom|opensubtitlescomvip|opensubtitlesvip|podnapisi|tvsubtitles]
                                  Provider to use (can be used multiple times).
...

I used https://pypi.org/project/entry-point-inspector/ to check that the entry-point was installed:
epi group show subliminal.providers

I've collected all the hacks I did to get my provider to actually run, check this link for details.

You don't need to use this hacks, see this example.


Revert "Don't generate internal protocolConfig for direct reach and f…

…requency measurement (#619)"
This reverts commit f407e899 until Kingdom is able to calculate direct reach and frequency

This change is

Hi @stevenwarejones. Thanks for pointing that out and you are right, it should be edp calculate direct reach and frequency instead of Kingdom. I was confused before. With that being said, I will implement the code in edp_simulator to calculate direct reach and frequency and fulfillDirectRequisition. I will open the PR for the change ASAP and this revert PR will not be needed


Fix engaged sessions

Description & motivation

Fix session_engaged event parameter. The value should be int_value, not string_value
Checklist

[x] I have verified that these changes work locally
[x] I have updated the README.md (if applicable)
[x] I have added tests & descriptions to my models (and macros if applicable)
[x] I have run dbt test and python -m pytest . to validate exists tests

@ivan-toriya-precis in the projects I have access to, I see session_engaged set as a string:

That's also what I see in this blog post here: https://www.ga4bigquery.com/sessions-dimensions-metrics-ga4/
Do you have any documentation from Google that is more concrete? Thanks for the PR!

@adamribaudo-velir
It's not documented or I just couldn't find in anywhere.
For the client I'm working with, it's in int_value, also double-checked with another one:

But, for some, I've found them sometimes in the string_value where platform = "WEB"
Weird, I'll try to investigate more about it.

Yes, I can look into that, but probably starting next week 👍

@adamribaudo-velir ready for review

for the enterprise client i work with, session_engaged can be in both int_value and string_value in a WEB only dataset:

So i coalesce both values:
(select COALESCE(value.string_value, cast(value.int_value as string)) from UNNEST(event_params) where key = 'session_engaged') as session_engaged,

I see you've done something similar in base_ga4__events, but with a CASE WHEN. Anyway it was just to add up on that topic :)


Add ApitTiken,Autopilot Api, To optimize the UpdateDeploymentDesiredS…

…tateParams,change the version number of Common to 1.0.10

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


{Util} Add util module for version and rest

Description
Peel off az version and az rest from resource module. A new module util is created. Thus they don't carry the burden of

azure-mgmt-resource initialization when running these commands
scenario tests on resource command module when developing these commands

This makes running and developing az version and az rest significantly faster.
Testing Guide
> az version
{
  "azure-cli": "2.6.0",
  "azure-cli-command-modules-nspkg": "2.0.3",
  "azure-cli-core": "2.6.0",
  "azure-cli-dev-tools": "0.1.1",
  "azure-cli-nspkg": "3.0.4",
  "azure-cli-telemetry": "1.0.4",
  "azure-cli-testsdk": "0.2.4",
  "extensions": {
    "application-insights": "0.1.1",
    "interactive": "0.4.4",
    "mesh": "0.10.6",
    "timeseriesinsights": "0.1.2",
    "webapp": "0.2.24"
  }
}

> az rest -m GET -u https://management.azure.com/subscriptions/0b1f6471-1bf0-4dda-aec3-cb9272f09590/resourcegroups/rg1?api-version=2019-07-01
{
  "id": "/subscriptions/0b1f6471-1bf0-4dda-aec3-cb9272f09590/resourceGroups/rg1",
  "location": "westus",
  "name": "rg1",
  "properties": {
    "provisioningState": "Succeeded"
  },
  "tags": {
    "StorageType": "Standard_LRS",
    "type": "test"
  },
  "type": "Microsoft.Resources/resourceGroups"
}

add to S171


(🎁) specify files in config file

I want to specify the "files" option in my config file, so that I can invoke like: 👉 autoflake.
[tool.autoflake]
files = ["scripts", "src"]

I did a draft to implement this functionality https://github.com/PyCQA/autoflake/pull/245
Also, when executing the pre-commit hooks, it noticed that ruff implements quite a lot of functionality of autoflake, and allows setting the src parameter in the config file.


Connectors: Official connectors need examples

This tracking issue will cover adding basic and full examples of connector configs. The goal is to demonstrate the options so they are more clear for how to use them. Preferably you can copy/paste them, and modify as needed to whatever value you want

Examples are included in the README docs as part of the connectors packaging, and our process embeds this documentation for reference


Polybase Fixes

Corrects issue with Polybase External Table statistics, which cannot be updated, but rather must be dropped and recreated.

Note this is to resolve:  https://github.com/olahallengren/sql-server-maintenance-solution/issues/532


What would a Data Science sku of VS look like?

I don't do Data Science ... I hear from lots of people I trust that F# is good for doing Data Science.  If we were to propose a packaging for VS for doing Data Science what would it need to have in it?
VS is extremely configurable, and it is possible to add high quality ... well supported 3rd party components into the dynamic installer feed.   That's how Xamarin for VS is delivered for example, although Xamarin is now 1st party.
I imagine that VS for Data Science would need:

F# enabled by default
F# Power Tools
FSharp.Data Type provider

What else?
Python?, R tools for VS?
I would like to see whether we can describe an ideal packaging of F# for DataScience, what we would need to add what samples are needed and what the documentation would look like?  Is this even a worthwhile activity?  should we expend our effort on something else?
Kevin

Better tooling for data science is definitely one of the goals of this Ionide work: https://github.com/ionide/ionide-fsi/pull/24, which lets you add HTML printers to FSI and embed outputs of commands as HTML.
Ultimately, the goal there is to enable nice integration with the FsLab charting libraries, R type provider and possibly other things (see www.fslab.org).

I agree with F# OOB with fslab
Having OOB R and Python language integration would also be good
Having some project templates for experiments (like on AzureML) in all 3 languages (F#, R, Python)
Having Accord.NET installed?
If you really want to go crazy, but the AzureML .dlls on the desktop too :-)

So, I figured out how to automate the installation of Visual Studio Code on a Docker image with VSCode extensions like Ionide installed. My first one is packaging a golang-vscode. It is almost working and I'll blog about it when it is, but I'd like to build one that targets F# for Data Science too. My intentions were to put together a dotnet-vscode with a collection of extensions from https://marketplace.visualstudio.com/vscode when dotnet core rc2 finally is released.

@ctaggart nice work.  @tpetricek Yes Ionide is indeed awesome... what does nice integration with fslab charting libraries and the R typeprovider look like.  Is it templates ... or samples or preinstall ...
@jamessdixon
So do the templates for experiments already exist?   What would productising them look like?
Is there any missing software ... pieces that just don't exist anywhere in the ecosystem?   or does it just need collecting together in a single packaging and productizing?
The goal here is to identify if there any actions we can take to put this together  ... perhaps come up with a plan and perhaps start work on it, although if VSCode + Ionide + Docker + FsLabs is sufficient then life is good and we can focus on something else that needs addressing.

I think the most blatant hole currently is having a way to easily create charts and visualize data. I am not entirely certain what the right solution/approach is at that point.

I vote a pre-installed environment with FSLab, Xamarin, Asp.net templates for F# and entity framework.  This would enable full stack development in F# with adhoc visualizations.  There is a need to have a ML library that is wrapped in an F# idiomatic way as well.  My vote is for CNTK or Tensor Flow.  Links to contribution pages as the default popup would be really nice (FSLab as well as training materials for newbies).
To support other languages R and Python, you would need to include whatever is available to deliver full products in those languages as well as perform primary workloads expected in those languages.  Example in R right click deploy to SQL would be pretty sweet.  Python would likely be flask (or django unfamiliar in python) templates and all packages that are common in those workloads.
Also consolidating the send to interactive button would be nice, be it ctrl-enter or alt-enter or whateve ryou want, as long as its only 2 buttons.

I'm obviously biased, and I'm not doing lot of data science but will allow myself to express opinion.
If you want to spend resources / time on Data Science tooling for F# I would go with making Ionide better instead of investing into VS.
Pros:

Cross platform support (what might be important given popularity of Macs in DS community [no stats, just my feeling from following stuff on web ])

Lightweight editor (VS is great for people coming from .Net / C# world, but lot of DS folks have different background. Forcing someone who is used to text editor / light IDEs to install big heavy IDE might be annoying. It's 100MB vs couple of GBs)

More flexibility ( VSCode and Atom let users to easily mix stuff, nothing stops anyone from having markdown, latex, F# scripts, R files , and Python stuff in one project)

Flexible extensions system ( especially Atom, but also VSCode have very good and flexible extensions APIs which lets us to really quickly add new features [like shown above by Tomas new FSI panel in Atom])

Community driven ( all used libraries which should be used [FSharp.Data, FsLab, Accord, MBrace etc ] are created by community anyways, we could go full ahead with it and also use editor driven by community)

Cons:

Project file UX ( we still struggle to provide good UX for handling fsproj files - right now they require by hand editing. But it can be change with integrating Forge)

Debugging story ( Atom doesn't support debugging at all [and would not expect it to get any extension API for that anytime soon ]. VSCode support F# debugging only on mono. It's fixable [by creating debugger adapter for MDbg] but it's not trivial)

Summing up:
From the beginning of Ionide me and Tomas have believed that data science is great niche for Ionide - it has been expressed by stuff like specific FsLab integrations or this new FSI panel which gives notebook style user experience. And we accept PRs ;)

@Krzysztof-Cieslak
I think VSCode could easily be a good basis for a data science toolkit based on F#.  I imagine that for the most part the list of things it needs would closely match VS.
Forgive my ignorance of the VSCode ecosystem, however, does anyone think that the ability to have access to in-IDE Python and a Python Repl is interesting, I know that VS has python tools for VS, I'm not familiar with VSCode?  Similar for R ... there is a VS integration of R ... would that be interesting for a Data science workbench?  and if so ... is there an equivalent tool for VSCode?
Or do we think that those tools are not useful?

inclusion of Jupyter and/or Beaker notebook with necessary addins to make F# and other languages usable there out of the box
revamp interactive REPL of all relevant languages to have a consistent experience and convenient printers (to display and inspect large arrays, dataframes, graphs)

It is true that many Data Scientists are used to lighter weight tools, but there is probably a good part of them who wouldn't mind having a well integrated tool with everything setup and streamlined experience.
http://jupyter.org/
http://beakernotebook.com/

I think from FsLab standpoint support in VS works already ok (if you don't use nuget since that one have sometimes issues to resolve the FsLab dependencies).
Things that are missing:

good visual data viewer
a way to have a live window that show results. IIRC there was no way to do this without an addin. So people went to other more powerful editor.
good graphics lib

Microsoft invested a HUGE amount of money into bringing R in VS. So maybe some of the excellent R Studio GUI tools will come as well.
All the other things are done in open source projects and FsLab handles orchestration and via paket the update of these tools/libs.

@smoothdeveloper regarding streamlined experience: that's why they move to jupyter and co - that's where actual scientists work and influence the design. VS is coming from a completely different background and doesn't really fit that model.
Every scientific project is different and many use a multitude of tools. Jupyter is one way to bring all that stuff into one environment. That's a great approach. It's not completely different mindset to type providers that allow you to integrate everything (well if we had more tps) into the fsx context.
So I think it's super important to build a jupyter kernel for F#. And the other way around to build the python TP. And even more important wrap all the Fortran libs from numpy and scipy into a lib rthat can be called from F#. That would make a huge difference.

I really like the idea of a Visual Studio Data Science sku featuring F#. I think an essential intermediary or foundational step would be to work on a Notebook/Workbook/Worksheet like experience for F#. Examples of this in other langs and/or editors include C# Workbooks in Xamarin Studio, Scala Worksheets in ScalaIDE/Eclipse, and of course, Jupyter.
Here are a couple of links to past conversations that have taken place about this subject:

https://github.com/ionide/ionide-fsharp/issues/104
https://github.com/fsprojects/VisualFSharpPowerTools/issues/185

Here is a link to the IfSharp project which was a F# profile for the old version of Jupyter, back when it was known as iPython Notebook:

https://github.com/fsprojects/IfSharp

Oh, the reason why I think it is an "essential intermediary or foundational step" is because it provides a general (not just Data Science) boost to developer joy and productivity by shortening the feedback loop. Just imagine what it would feel like working with F# Data, SwaggerProvider, RProvider, Deedle and/or Spreads, and MBrace (all of which is already quite compelling on good ol' FSI). Your work would be almost entirely limited to modeling and algorithmic design/analysis! This in an type-safe, expressive, functional language!!!

@lambdakris regarding your last comment - where do you think would VS it into that picture?

@forki Well, really it is just my attempt to express why I think a Visual F# Notebook-like experience is to be emphasized before other Data Science supporting features for VS. Ultimately a Visual F# Notebook experience in Visual Studio is what I'm advocating.

Folks,
I think that VS would be beneficial because it would bring what is needed together into one package that many Enterprises would consider installing.  Support for more than one language is beneficial because DS today is a polyglot endeavor. Having a consistent, low friction, polyglot, trusted packaging would enable documenters to create straightforward usage instructions for a number of common scenarios, and would enable community minded data science focused tools developers a vehicle to target their Data Science solutions, algorithmic libraries and services.
The idea is to pick something and make it great for the DS scenario, I know @mathias-brandewinder and @tpetricek have been advocating something like this for several years.  But it has been quite slow to come together.  I'm throwing the VS hat in the ring because it has a number of advantages related to acceptance and adoption --- but I am perfectly happy to support any alternative formulation --- wherever we go with this, we should all get behind it and push.
Kevin

My wish list:

Cross platform off the bat; Atom with Ionide preferably because I can write my other languages in that editor.
Rewrite the Process API so I can call into Python without first shaving five yaks. Immutability please.
TensorFlow F# API with "extra features" like aleaGPU integration as custom activities (but please don't buy them, just integrate with them and cooperate in an OSS manner). CNTK has absolutely no uptake.
The F# Jupyter integration works really well already; it would be great to have it better featured.
Boost the fsdot nuget so I don't have to maintain it
Help maintain and improve Accord.Net, Tesseract.NET, Stanford NLP and FsLab
Write excellent docs that users can contribute to. Take after how google does it with TensorFlow – really excellent docs that get you started
Release things done on homebrew proper. No custom packages like coreclr and no custom taps. Work with the ecosystem.
Please, no more Sql type providers
Feature the F#-native and idiomatic libraries. Build libraries, not libraries into tooling. Let your users drive progress by composing libraries. Open-ended systems over Closed systems.

I don't think that F# devs are lacking more tooling. They are lacking cohesion in story of F# Data Science.

@KevinRansom The problem isn't about Visual Studio or whatnot - having a good VS story will only get us so far. If Microsoft wants to help with the DS story, F# needs (as @haf rightly says) a coherent end-to-end story. This includes a modern ML library with good examples (ideally one that is distributable). More investment on Deedle. Having MS help build confidence in the developer community that, yes, you can use .NET for data science - currently I see MS pushing and investing in R and Python - both languages that have good DS stories to a point - but I don't get the impression that there's much awareness of the capabilities of F# for this story as well, as well as within Azure. The list goes on and on.
So yes, whilst getting some templates in VS would definitely be useful, tooling is just one element of this story - it can't be solved just with that.

A useful offering would address a broader array of data science tasks.  The scientific quantitative analysis enabled by the variety of tools mentioned above is usually the penultimate step in the acquire-sanitize-normalize-analyze-proselytize data science project cycle.  Often the most time-consuming and expensive part of the process lies in the first three, or the second and third steps.  The analysis and presentation work is often done by a smaller team of specialists and is rarely the binding constraint when planning/budgeting a data analysis project.     A few suggestions:  (1) Acquisition .. include built in  processes for a wide variety of published data, along the lines of the F# data providers, and /or specialized R packages. For finance and economics links to FRED, Quandl, the IMF and World Bank, O.E.C.D. and several U.N. Agencies would be popular,  for text-based analysis access to dictionaries and thesauruses (e.g. WordNet) ,  for merchandising access to Census decennial and other surveys. , Of equal importance would be the ability to easily import intermediate and final results from other systems, including but not limited to S,SPSS,SAS, Matlab, Wolfram, Julia,..et al ..... Sanitation....a.k.a. data munging.... provide illustrated guides and walk-throughs for exploring data and detecting and ameliorating errors and omissions.  (shortcomings here always lessen the final product).  Normalization...This is the lead in to the analysis.  Each combination of subject area and analysis method impose distinct requirements.  Here detailed examples and templates, specifically targeted to subject areas and methods  .    The great potential for a VS offering would be the ability to incorporate prior efforts and results by allowing the polyglot projects with R, Python, F# and other .NET modules.....

Microsoft should really put efforts and push F# as the language of choice for Data Science (within Microsoft's eco-system) and scaling algorithms in the cloud (mbrace) and parallelization (hopac, streams):

it combines all the advantages of statically typed languages while enabling same overal looking code as with dynamic languages *
it has demonstrated ability to unify disparate and data-rich environment with use of Type Providers
outstanding support for scripting and REPL from day one (which is a tenet to loading dataset and working interactively with it)
interop with .NET

Dynamic languages are most commonly used in Data Science, and it causes lots of pain, I know people doing this type of work, and updating a library often involves hours of troubleshooting in the code due to lack of compile time checks, one issue at a time.
Static typing gets in the way mostly when you have to state everywhere the types once or twice (like in C# and VB.NET, and this gets really noisy as soon as you have the slightest amount of generics).
Time for Microsoft to realize and invest efforts in F# making :

a killer FSI
a killer set of Type Providers and wrappers for most commonly used tools in Data Science
outstanding experience for visualization and skimming through large datasets within VS

That is a good way to acquire a lot of Data Scientists as Visual Studio users :smile:

I work in economics, and a lot of people use RStudio or Stata, sometimes together with Python, for data analysis. These tools have their strengths and weaknesses, but as a language, F# is lightyears ahead of them, especially with the VS integration for intellisense, syntax checking, debugging, etc. I could imagine F# in Visual Studio replacing RStudio or Stata (the ability to access R via type providers is pretty big), but it has to be adapted to the right workflow.
Economists working with RStudio or Stata have a very different workflow from programmers. They don’t sit down and start writing code from a project template (much less from scratch). They open Stata or RStudio and start pointing and clicking, typing interactive commands into the REPL or a combination of the two. It starts with loading/downloading data, and doing exploratory analysis, which is gradually refined over time and eventually consolidated into scripts that are used to load and clean/label data, to run statistical analyses, to generate tables and figures in various formats, etc.
Probably the single best feature of Stata is that it has a rich GUI that can be used almost like Excel, as well as a Command window, and both interfaces interact with the same REPL core. Whether you type a command into the Command window or point and click, the resulting command line (as typed or generated via the GUI) is printed in the output window and logged in the Review window. Beginning users who prefer to point and click can often avoid the REPL entirely, but the results are exactly the same, and can be exported as a script.
The Review pane contains a history of every command executed in the session, but not output. Any command in the history can be re-run with a double-click, and part or all of the history can be exported as a script (‘do file’), which can be run by any other Stata user. That means that everything a user does in a Stata session can be exactly replicated by anyone else. This is really important. A GUI user can even open the Excel-like Data Editor and start typing values into cells, and every single entry will be translated into Stata commands that appear in the output and in the Review pane. The value that adds is enormous compared with, say, Excel, where you never really have any idea what someone has done, or a typical IDE like VS where you really have to start by writing code, creating a barrier to non-programmers coming from, say, Excel.
I think a Data Science version of VS (VS/DS) would have to be centred on a REPL, in much the way that Stata is. VS has the Command Window pane, but it’s pretty useless. You have to type long, tedious menu commands rather than concise commands, and it really feels like an afterthought. To my knowledge, there isn’t even a way to save the session history. Moreover, activity done with the GUI is completely separate. It’s chaotic compared with the unified environment offered by Stata.
Rather than trying to improve the existing Command Window in VS, it would make much more sense to make an enhanced FSI the heart of VS/DS, with F# libraries exposing all of the features of the VS GUI, especially those related to data analysis. These libraries could be loaded automatically at startup, allowing command-line users to ignore the GUI entirely and just start typing into the REPL (which is what more experienced Stata users typically do).
Following Stata, I think FSI should be split into three panes/windows, an input pane, an output pane and a review (command history) pane. Ideally, that would be combined with GUI hooks, like in Stata, so that anything that is done with the GUI would be translated into FSI commands and appear in the output and review panes. The result would be an environment with a low barrier to entry, helping beginners get started, but one where the GUI wouldn’t get in the way of experienced users, or those with programming backgrounds.
TLDR: Look at how the Stata (http://www.stata.com/) GUI and REPL are linked together, and build something similar in VS, with an enhanced FSI as the REPL at the core.

It would be very nice to have a FSEye style object monitor built into VS & FSI so you can always monitor objects currently in memory and potentially have context menus to chart/export. When working in MATLAB/Octave, you are always able to see all your currently loaded objects/values and this is really useful for Data Science, FSEye kind of does this but to have it built in with extensive functionality would be great.

@gerardtoconnor on same ground about monitoring objects, make debugger visual inspectors available in F#, they only work in C# and VB.NET for some reason.

A quick thought: one of the ways to look at the question is, which 'style' do people want? I can see two existing approaches:

RStudio: a scripting/REPL environment, with a rich output window where all charts & outputs go,
Notebooks: a single window, alternating inputs and rich outputs.
IFSharp (https://github.com/fsprojects/IfSharp) already offers an implementation of (2). Visual Studio is currently closer to (1), but the output window (FSI) is extremely crude. What @tpetricek showed with Code+Ionide is close to what I prefer, but I am not sure how this would look in VS.

We need to bribe @nosami to bring xamarin F# repl to VS ;-)

@mathias-brandewinder I prefer a REPL/scripting environment, ideally with the input pane separate from the output pane(s)/window(s), as in Stata (command lines are still echoed to the output pane, but not typed into it, with other types of output going to other panes/windows) but not RStudio (text output appears in the same pane you type commands into, but other output appears in other panes).
With the REPL/scripting approach at the core, it probably wouldn't be too difficult to allow different settings for input and output. You could do everything in an HTML-based 'notebook' pane that sends commands to the REPL and then displays the rich output inline, or have separate panes for input, text output, documentation, HTML output, plot output, etc. The critical thing is that everything should ultimately be translated to commands sent to a REPL (in the open or behind the scenes), so the session (not just the final output) can be saved and replicated.

@forki on this note, I need a way to surface my data science into modern applications, I'm trying to do this through Xamarin, however there are a few key items that I am finding don't work.  Investments in FSLab to ensure everything exposed there can run in Xamarin and .Net Core would be incredible.
Beyond making FSLab work in Xamarin/.net core, I think the power of data science languages comes from the packages, primarily the visualization, data ingestion/manipulation and machine learning frameworks provided.  I think to be truly successful, the major thing we are lacking in for the Microsoft platform is hitting modern app targets with all of those peices.  The tooling is something that can be worked on, but dude, I can't use my code on device, because most if not all of the libraries are not able to target these things.
I think if those baseline packages can really get nailed in a good way, you have the framework for building and delivering amazing products which have data science at the core of them.  This is the direction I believe the industry is going to go and is the way I am moving, but I find myself blocked at various points because of these limitations.
For the tooling, I think most everybody is on par with what I'm after as well, great REPL experience, various windows and plots, but important for me also, is being able to easily convert that into a production library that can be consumed by a modern client application.
I'm of the opinion the tooling is in a decent place right now, the gap is the libraries and the platforms they can target.

I'm of the opinion the tooling is in a decent place right now, the gap is the libraries and the platforms they can target.

Agree with @drcrook1 on this.

@KevinRansom
Python support in VSCode is top class, don't know about R... but not sure if we should focus on R given R TypeProvider.
The question is: If you want to create SKU for VS (as in title), or good F# Data Science experience in general. I will focus on second case (as I believe it's better for F# in general). In such case locking in VS (and thus Windows) is very bad idea. There are many place you could contribute - x-plat editors (arguably, that's easiest thing and community can handle it) and whole set of libraries (others people in this thread are more qualified to say exactly what should be improved)
As others mentioned it's also important decision what style of editor we want for DS (or maybe we want / need all?) - inlining output in editor, rich REPL where you can type, or maybe something like what we have currently in Atom + Ionide (read-only [but rich] FSI panel and communication only by send line/selection/file)

@Krzysztof-Cieslak
To what extent do you think improving the DS experience could be done with F# libraries that could be used in both VS on Windows and VSCode/Atom on other platforms? I assume VS uses XAML for UI, whereas the others use HTML, but if all of the underlying code is F# without dependencies Win32-only libraries, maybe it would be possible for DS libraries that deal with UI to support both kinds of rendering?
Both a lightweight, cross-platform experience based on VSCode, and a more advanced, heavyweight experience based on VS seem useful. As a VS user, I have to say that Atom has a long way to go to be comparable in terms of features, usability, performance and robustness, and VSCode has even further to go in many areas (text rendering on Windows is so bad as to be barely useable – I don't know about on other platforms). If VSCode could match essential features, usability and robustness of VS, and remain lightweight and cross-platform, it would be great complement to VS, but I think both will remain useful.
I really like the way Ionide is going, and hope the VSCode experience can eventually catch up with the Atom experience. I also agree that it's important for the F# community that things are open source and cross-platform.  If it offered the same sort of REPL-centric experience as RStudio or Stata, coupled with its current capabilities and the ability to target all the major platforms, plus access R packages, it could be an amazing DS tool.

Lots of good feedback here so I won't repeat what everyone else says, just a quick comment that there's a functioning Jupyter branch in the official IFSharp repo.

As I mentioned earlier, I like the idea of bundling a bunch of extensions and software with VSCode in a Docker image. You can run the image on Windows, Linux, Mac, in Azure or another cloud, or your own datacenter. The shell and GUI are still accessible locally no matter where you run it. It should take less than 10 minutes to set it up. Here is a video I created demonstrating the same concept with Rust extensions. It runs the image on a machine on Azure, but displays it locally on my Windows 10 laptop. https://twitter.com/cmr0n/status/729508355144716288
With VSCode, we can put ship F# again in the box:

That is a screenshot of running the first build from https://github.com/ctaggart/dotnet-vscode. All those extensions get prepackaged.
I'm not making any promises about the current stability of dotnet-preview:latest. I'm really looking forward to RC2!
Here are my quick notes (a blog is coming) about running this with Docker for Mac:
in one terminal tab
docker run --rm -it -p 6100:6100 ctaggart/dotnet-vscode
. start-xpra.sh
code .

in another terminal tab
export PATH=/Applications/Xpra.app/Contents/MacOS:$PATH
xpra attach tcp:<IP_ADDRESS>:6100

Just want to do small update with our current state of the art in Ionide

Source: https://twitter.com/fslaborg/status/728397249697488896/photo/1
@tpetricek can give us more details about it.
It's currently working in Atom (as Tomas is using it) but I'm fairly confident we should be able to port this to VSCode.

I love the great progress! I'm really hoping that can be ported to VSCode. In VSCode, I like that their debugging support and their openness with the public iteration plan linked from their insiders download page.

@ctaggart We're always looking for contributors :-). It is mostly a matter of porting the work from ionide-fsi (which is for Atom) to ionide-vscode-fsharp (which is for VS Code). Open an issue there to discuss what needs to be done!

How does that play with the jupyter notebook project?

@drcrook1 interesting that you mentioned that, because last week at MSR we had a good look at Jupyter with F# support - it's progressed quite far now and I'm hopeful that it will become a "mainline" branch very soon (https://github.com/fsprojects/IfSharp).
@ctaggart - @Krzysztof-Cieslak is also close to getting the Atom stuff into VSCode now. If only @tpetricek would fix the binding redirects :-P

Can somebody send me a note the second you publish the VS Code and Jupyter goes mainline.  Maybe shoot me some docs on the Jupyter notebook?  I'm running an ML study group that uses F# and would love a lower barrier to entry and better experimentation framework for teaching.
Also, let me know what I can assign my students on.  I should have ~3 students starting in August who will need to contribute to repos.

@isaacabraham This is the change that breaks everything. I have no idea why. Will try to figure out tomorrow, but if anybody wants to help... :confused:

Yeah, it's a weird one. I did manually copy over the correct FSharp.Core, and it complained spectacularly when I didn't put in the correct BR - but then when I updated them to <IP_ADDRESS>, the errors went away but still complained when sending to FSI about missing methods e.g. sortByDescending etc..

Closing old discussion


use staticcheck instead of deprecated megacheck

And here I was wondering if anyone was actually using this anymore. I'll go ahead and merge this for now, but I definitely need to get this switched over to Go Modules and give the code a once-over. Cheers.

Went ahead and did a quick cleanup on my end. Enjoy!

:+1:


Add Rake Task for Removing Duplicates

Context
Trello
Staffordshire University have sent over a spreadsheet of around 70 trainees who have duplicate records in Register. The duplicates need to be deleted.
Changes proposed in this pull request
Added a rake task to search for the duplicates based on the trainee's email address and discard the duplicates.
Usage
For a CSV file in the following format:
trainee_id,email
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
The rake task can be called like this:
 bundle exec rake trainee:remove_duplicates [path_to_csv]
Important business

[ ] Does this PR introduce any PII fields that need to be overwritten or deleted in db/scripts/sanitise.sql?
[ ] Does this PR change the database schema? If so, have you updated the config/analytics.yml file and considered whether you need to send 'import_entity' events?
[ ] Do we need to send any updates to DQT as part of the work in this PR?
[ ] Does this PR need an ADR?

NB: Please notify the #twd_data_insights team and ask for a review if new fields are being added to analytics.yml

Thanks for putting some instructions in the PR description. I think it would be useful to have these in the app docs as well, so they're easy to find for future use. I reckon a section in docs/support_playbook.md would be good as this could be a regular task which falls to support.

Ah sorry, I thought this would be a run-once thing, but then I remember you saying this could stick around. I'll update 👍

Updated here @tomtrentham

 Duplicate records - Staffordshire University