Added version update notification

Adds notification of version update availability
🗣 Description
This pull request resolves #1285 by introducing version update checks at import time, which are enabled by default. A new environment variable, SCUBAGEAR_SKIP_VERSION_CHECK, enables users to skip version update checks. Setting this option to any non-whitespace value disables version checks on module import. The checks will target both PSGallery and Github in an attempt to determine whether a newer release is available. If a new release is available, a warning is printed to the console.
💭 Motivation and context
Users get the best possible assessment by using the latest version of the ScubaGear tool, with the latest baselines. Later versions incorporate updates, fixes, and enhancements. A mechanism to notify users of a new version when running ScubaGear.
🧪 Testing
I tested these changes by modifying the ScubaConfig module and verifying that:

The SCUBAGEAR_SKIP_VERSION_CHECK  env var enables or disables checks when set in the config file
When scuba is installed from PSGallery, if the version is not the latest version, the warning message is printed to the console. If the installed version is the latest version, no warning is printed.
When scuba is installed  from Github, if the current revision is not a tagged release, the warning message is printed to the console; otherwise no message is printed.

EDIT: Made changes to reflect the change from a command line switch to environment variable.
✅ Pre-approval checklist

[x] This PR has an informative and human-readable title.
[x] PR targets the correct parent branch (e.g., main or release-name) for merge.
[x] Changes are limited to a single goal - eschew scope creep!
[x] Changes are sized such that they do not touch excessive number of files.
[x] All future TODOs are captured in issues, which are referenced in code comments.
[x] These code changes follow the ScubaGear content style guide.
[x] Related issues these changes resolve are linked preferably via closing keywords.
[x] All relevant type-of-change labels added.
[x] All relevant project fields are set.
[x] All relevant repo and/or project documentation updated to reflect these changes.
[x] Unit tests added/updated to cover PowerShell and Rego changes.
[ ] Functional tests added/updated to cover PowerShell and Rego changes.
[x] All relevant functional tests passed.
[x] All automated checks (e.g., linting, static analysis, unit/smoke tests) passed.

✅ Pre-merge checklist

[x] PR passed smoke test check.
[ ] Feature branch has been rebased against changes from parent branch, as needed
[ ] Resolved all merge conflicts on branch
[ ] Notified merge coordinator that PR is ready for merge via comment mention
[ ] Demonstrate changes to the team for questions and comments.

✅ Post-merge checklist

[ ] Feature branch deleted after merge to clean up repository.
[ ] Verified that all checks pass on parent branch (e.g., main or release-name) after merge.

@adhilto I added tips into the configuration and parameter docs so people know how to silence the version check warnings.

@twneale Looks like there's a minor version conflict here.

Some thoughts about the Invoke-CheckScubaGearVersionGit. Obviously, this function depends on Git. But I don't think that dependency is needed. Draw-backs of that method:

If a user installs ScubaGear by downloading the release zip from github.com, they may or may not have git installed
If a user installs ScubaGear by downloading the release zip from github.com, even if they have git installed, git describe --exact-match --tags returns fatal: not a git repository..., so the user gets the warning WARNING: ScubaGear is not currently at a GitHub tagged release... I don't like that one of our "official" ways of installing ScubaGear will always produce a warning.
If a user installs ScubaGear via git clone and stays on main, they'll still get the warning, because git describe --exact-match --tags returns fatal: no tag exactly matches... The git method will only work if the users do something git checkout v1.4.0, which isn't something we're actually telling our users to do anywhere.

But the git dependency isn't even needed, as it's just used to determine the currently installed version, and that information is stored in the module manifest file. For example, you could do something like this:
$ScubaManifest = Import-PowerShellDataFile (Join-Path -Path $PSScriptRoot -ChildPath 'ScubaGear.psd1' -Resolve) -ErrorAction 'Stop'
$CurrentVersion = $ScubaManifest.ModuleVersion
$LatestVersion = $(Invoke-RestMethod -Uri "https://api.github.com/repos/cisagov/ScubaGear/releases/latest").tag_name.TrimStart("v")
if ($CurrentVersion -ne $LatestVersion) {
    Write-Warning "A new version of ScubaGear ($latestVersion) is available. Please consider updating at: https://github.com/cisagov/ScubaGear/releases"
}

That should work if:

Git isn't installed
You install by downloading the zip file
You install by cloning the repo

Thoughts?

Some thoughts about the Invoke-CheckScubaGearVersionGit. Obviously, this function depends on Git. But I don't think that dependency is needed. Draw-backs of that method:

If a user installs ScubaGear by downloading the release zip from github.com, they may or may not have git installed
If a user installs ScubaGear by downloading the release zip from github.com, even if they have git installed, git describe --exact-match --tags returns fatal: not a git repository..., so the user gets the warning WARNING: ScubaGear is not currently at a GitHub tagged release... I don't like that one of our "official" ways of installing ScubaGear will always produce a warning.
If a user installs ScubaGear via git clone and stays on main, they'll still get the warning, because git describe --exact-match --tags returns fatal: no tag exactly matches... The git method will only work if the users do something git checkout v1.4.0, which isn't something we're actually telling our users to do anywhere.

But the git dependency isn't even needed, as it's just used to determine the currently installed version, and that information is stored in the module manifest file. For example, you could do something like this:
$ScubaManifest = Import-PowerShellDataFile (Join-Path -Path $PSScriptRoot -ChildPath 'ScubaGear.psd1' -Resolve) -ErrorAction 'Stop'
$CurrentVersion = $ScubaManifest.ModuleVersion
$LatestVersion = $(Invoke-RestMethod -Uri "https://api.github.com/repos/cisagov/ScubaGear/releases/latest").tag_name.TrimStart("v")
if ($CurrentVersion -ne $LatestVersion) {
    Write-Warning "A new version of ScubaGear ($latestVersion) is available. Please consider updating at: https://github.com/cisagov/ScubaGear/releases"
}

That should work if:

Git isn't installed
You install by downloading the zip file
You install by cloning the repo

Thoughts?

I agree this way is better. I blew away the git checks in favor of looking up the ModuleVersion in the manifest 👍

@nanda-katikaneni This branch should be ready to merge 👍


Fix note indentation since it's causing the CSS to break.

This is a fix for https://github.com/kubernetes/website/issues/20902

/assign

/assign @sftim

/language en

/assign @jimangel

/unassign

@irvifa
If you have a PR open, there is a rota (mainly volunteers) of people who review PRs for the main English localization, and other localizations take a similar route.
You usually don't need to assign to a particular person to request reviewing; the act of opening a pull request counts as asking for a review from whoever is free first.

@sftim any possibility of checking previous versions of the hugo? Many pages is broken as well. And it’s not comfortable to type in such a long lines isn’t?

@irvifa I'm not sure what you're asking there; can you clarify?

So basically I notice that there’s changes in the netlify hugo version. I’m not sure when does this change. But for the pst few days I’m encountering several error and I suspect the cause is just the same. Meaning that within the note brackets we can enter a new line. So this is kind of uncomfortable because we can’t separate a long paragraph into several lines within the block.

@irvifa please do feel welcome to log issues about bugs that you see. The change you're thinking of was PR #19907 that upgraded Hugo to 0.70

@sftim turned out I need to change the template for note class. I mentioned it in the new PR.


Adding prettier

Hey folks. I think it's worth adding prettier for code formatting. What do you think?
https://prettier.io/

Not really a fan + we already have a few linting rules that help with the formatting

@FokinAleksandr What do you think it will bring to the project? What do you see in the code that prettier would make it "prettier"?

@FokinAleksandr What do you think it will bring to the project? What do you see in the code that prettier would make it "prettier"?
For me personally the biggest benefit is that you don't have to think how to align your code at all. No need to spend time pressing tabs, spaces, enter. You just rapidly throw some code on 1 line, press the button and everything is aligned consistently with the rest of the project.
I do understand thought that this tool is pretty opinionated and formats code in a certain way without too many options to customize it.

Right now that is made consistent with the linter rule. And it is fixed automatically using the fix command of the linter. Rarely it complains about some nested blocks and you have to fix it manually, but doesn't happen often. So that part of the issue is solved already, isn't it?

Right now that is made consistent with the linter rule. And it is fixed automatically using the fix command of the linter. Rarely it complains about some nested blocks and you have to fix it manually, but doesn't happen often. So that part of the issue is solved already, isn't it?

Yeah but since it's not as strict you still have to spend time thinking how to format your code and spend time pressing tabs, enters, spaces and whatever. That's the main difference.

Thank you for the explanation. I still think it is not needed, so I am tempted to close this ticket as a "won't do". Nevertheless, let's leave it for a week or two open, in case anyone else wants to chime in to try to convince. What do you think?


rust version error

 0: Mismatched rust versions: `cargo-pgx` was built with Rust `1.68`, but Rust `1.67` is currently in use.

My rust is version 1.68, but plrust requires 1.67. Can I make plrust consistent with the installed rust in the system?

The version info:
[root@VM-0-14-centos plrust-1.0.0]# rustc -V
rustc 1.67.1 (d5a82bbd2 2023-02-07)
[root@VM-0-14-centos plrust-1.0.0]# cd /opt
[root@VM-0-14-centos opt]# rustc -V
rustc 1.68.2 (9eb3afe9e 2023-03-27)

Unfortunately no, even if you got past that error, PL/Rust uses a custom fork of the standard library (https://github.com/tcdi/postgrestd) and a custom Rustc compiler driver (https://github.com/tcdi/plrust/tree/main/plrustc). So it has a hard dependency on 1.67.1.

@thomcc Thanks.I will keep an eye on it because I want to use my own installed PostgreSQL and rust, but plrust and pgx prefer to have some built-in software, but that's not what we expected. We hope to develop stored procedures and extensions for the existing PostgreSQL on this machine

You should be able to use your own PostgreSQL, just not your own Rust. I don't think we have plans to change that (and am not sure how we would).

PL/Rust is compatible with Postgres v13, 14, 15.
Building PL/Rust, and then the resulting LANGUAGE plrust functions, do, however, require the Rust 1.67.1 toolchain.
As time goes on we'll likely update toolchain requirements, but we'll always be some bit behind as we have our own fork of the standard library (postgrestd).


Fix imports and backport to 1.3.3

Fix all the imports to reference this repo, and bring in textflag.h which is necessary to build on 1.3.3

+1


argcomplete + configargparse: adding certain types of config directives to config files breaks tab-completion

I LOVE argcomplete. I also love configargparse, a drop-in replacement for the standard argparse which allows you to integrate config files & environment variables with standard cmdline opts.
Thankfully they work together almost flawlessly.
Almost because when I put certain types of options in a config file argcomplete totally fails. It doesn't complete ANYTHING.
Here's a reproducer:
$ cat koalaberz.py
#!/usr/bin/python
# -*- coding: utf-8 -*-
# PYTHON_ARGCOMPLETE_OK
try:
    import configargparse
except:
    print("Missing configargparse; Run:\n  pip install configargparse\n")
    raise
try:
    import argcomplete
except:
    print("Missing argcomplete; Run:\n  pip install argcomplete; activate-global-python-argcomplete\n")
    raise

p = configargparse.ArgParser(default_config_files=['koala1.cfg','koala2.cfg'])
p.add('--togglefoo', action='store_true', help='Turn on the foo')
p.add('--appendbar', action='append', metavar='THINGS', help='Add things to bar')
p.add('--pickbaz', choices=['a', 'b', 'c'], help='Choose your baz')
argcomplete.autocomplete(p)
opts = p.parse_args()
print(opts)

$ chmod +x koalaberz.py

$ ./koalaberz.py -h
usage: koalaberz.py [-h] [--togglefoo] [--appendbar THINGS]
                    [--pickbaz {a,b,c}]

Args that start with '--' (eg. --togglefoo) can also be set in a config file
(koala1.cfg or koala2.cfg or ) by using .ini or .yaml-style syntax (eg.
togglefoo=value). If an arg is specified in more than one place, then command-
line values override config file values which override defaults.

optional arguments:
  -h, --help          show this help message and exit
  --togglefoo         Turn on the foo
  --appendbar THINGS  Add things to bar
  --pickbaz {a,b,c}   Choose your baz

Look good? Now a quick demo that the tab-completion works.
$ ./koalaberz.py -                                                           <--- Pressed Tab here
--appendbar  -h           --help       --pickbaz    --togglefoo
$ ./koalaberz.py --pickbaz                                                   <--- Pressed Tab here
a  b  c
$ ./koalaberz.py --pickbaz a --togglefoo -                                   <--- Pressed Tab here
--appendbar  -h           --help       --pickbaz    --togglefoo
$ ./koalaberz.py --pickbaz a --togglefoo --appendbar junk --appendbar more   <--- Pressed Enter here
Namespace(appendbar=['junk', 'more'], pickbaz='a', togglefoo=True)

When I add config directives of type store, all is well.
$ echo pickbaz = b >koala1.cfg

$ ./koalaberz.py -                                                    <--- Pressed Tab here
--appendbar  -h           --help       --pickbaz    --togglefoo

$ ./koalaberz.py                                                      <--- Pressed Enter here
Namespace(appendbar=None, pickbaz='b', togglefoo=False)

But watch what happens when I add a boolean:
$ echo togglefoo = true >koala2.cfg

$ ./koalaberz.py                                                      <--- Pressed Tab here
.bash_history  .bashrc        .config/       koalaberz.py   .ssh/
.bash_logout   bin/           koala1.cfg     .lesshst       .viminfo
.bash_profile  .cache/        koala2.cfg     repro          .vimrc

$ ./koalaberz.py                                                      <--- Pressed Enter here
Namespace(appendbar=None, pickbaz='b', togglefoo=True)

So adding a boolean toggle to a config file breaks argcomplete for some reason. ??? Note that configargparse lets you specify booleans in multiple ways, but none of them work with argcomplete.
$ echo togglefoo >koala2.cfg
$ ./koalaberz.py
Namespace(appendbar=None, pickbaz='b', togglefoo=True)

$ echo --togglefoo >koala2.cfg
$ ./koalaberz.py
Namespace(appendbar=None, pickbaz='b', togglefoo=True)

$ echo togglefoo: true  >koala2.cfg
$ ./koalaberz.py
Namespace(appendbar=None, pickbaz='b', togglefoo=True)

Another oddity is that options of type append work as long as I don't actually store more than one value in the list.
$ echo appendbar = barbar >koala2.cfg
$ ./koalaberz.py
Namespace(appendbar=['barbar'], pickbaz='b', togglefoo=False)
$ ./koalaberz.py -
--appendbar  -h           --help       --pickbaz    --togglefoo

$ echo appendbar = [bar 1, bar 2] >koala2.cfg
$ ./koalaberz.py
Namespace(appendbar=['bar 1', 'bar 2'], pickbaz='b', togglefoo=False)
$ ./koalaberz.py                                                      <--- Pressed Tab here
.bash_history  .bashrc        .config/       koalaberz.py   .ssh/
.bash_logout   bin/           koala1.cfg     .lesshst       .viminfo
.bash_profile  .cache/        koala2.cfg     repro          .vimrc

Given how awesome both of these modules work together ... and how I hope they'll be in the standard library someday, I'd love to see this fixed, but of course it's an odd problem ... what do you think? Knowing the argcomplete code like you do ... do you have any ideas of what's causing this?
Thanks for reading,
Ryan

Sorry about the lengthy delay in my response to this. I unfortunately don't know enough about configargparse to figure out why this happens. Ultimately I'm surprised argcomplete works with configargparse to any extent at all. Beacuse argparse is not actually built to allow customization of the argument parsing process, argcomplete relies on runtime monkey-patching of the parser (conditional upon detecting that argument completion has been invoked). Of course this monkey-patching has no API contract to stand on, so it's brittle and will break if you replace the argparse parser with something else.
I'll try to read more into what configparse does to see if argcomplete can be more defensive. If you want to give it a try, see the "Debugging" section in the readme.

Bumping this up in my queue, since I've seen configargparse used out there a few times now. Any help welcome.

Both issues are resolved by https://github.com/bw2/ConfigArgParse/pull/78. We don't need to make any code changes, but it might be a good idea to add a test that verifies the examples continue to work as expected.

The boolean issue appears to have resolved itself somehow, but having multiple options of type append is still broken.

@ryran I've followed through your examples with ConfigArgParse 0.12.0 and they work as expected. Thanks for the detailed report.


Fix kube-apiserver certificate when DNS is disabled

How to categorize this PR?
/area quality
/kind bug
/priority normal
What this PR does / why we need it:
When DNS is disabled, the kube-apiserver certificate should contain the LoadBalancer address of the kube-apiserver service. This is broken with the current flow, I was able to reproduce it a few times in our OpenShift test cluster.
Which issue(s) this PR fixes:
Fixes #
Special notes for your reviewer:
The fix introduces a dependency from the loadSecrets task to the waitUntilKubeAPIServerServiceIsReady in the reconciliation flow when DNS is disabled (or rather, moves that dependency from generateSecrets to loadSecrets). I tested the fix in our OpenShift test cluster and I wasn't able to reproduce the issue with it.
Release note:
NONE

/cc @plkokanov @rfranzke

Can you explain how this PR is related to https://github.com/gardener/gardener/pull/2643? Did I miss a case in #2643?

@rfranzke I think the explanation is basically the same that you wrote in that PR. The LoadExistingSecretsIntoShootState actually generates the secrets (the call to the secretsManager Load method actually calls the secrets generator). Regarding how exactly this leads to the issue, @plkokanov could provide more information. What I am sure is that I reproduced the issue a few times (by deleting the secret from both the shoot namespace and the shoot state, and then regenerating), and with the proposed fix everything worked fine.

/assign

The loadSecrets step will call wanted_secrets.go:110
I've been trying to see whether this somehow affects the DNSNames for the kube apiserver secret config, but have yet to find out how.

/status author-action
@stoyanr We have checked the code some more and can't find the place where the loadSecrets task would generate the server certificate for the KAPI of the shoot.
Can you share more about

how you produce this issue with the current master
why you think that these changes fix the problem?

@rfranzke I was able to reproduce the issue several times on Monday but not today. It's not reproducible every time, since it's basically a race condition between waitUntilKubeAPIServerServiceIsReady and loadSecrets.
How to reproduce:

Stop gardenlet
Delete the kube-apiserver secret from shoot state and from the shoot namespace
Start gardenlet again, get the certificate from the secret, and decode it with openssl

Just a note, I edited the reconcile flow to always execute the loadSecrets step before waiting for the api service loadbalancer to be ready (just for testing purposes). Here are the entries in the api-server cert for subject alternative names
DNS:kube-apiserver, DNS:kube-apiserver.shoot--mig--migration1, DNS:kube-apiserver.shoot--mig--migration1.svc, DNS:api.migration1.mig.internal.dev.k8s.ondemand.com, DNS:kubernetes, DNS:kubernetes.default, DNS:kubernetes.default.svc, DNS:kubernetes.default.svc.cluster.local, DNS:ab33251787fa1404e803de99c4f523c8-1366641310.eu-west-1.elb.amazonaws.com, IP Address:<IP_ADDRESS>, IP Address:<IP_ADDRESS>

I will run it a couple more times and move the loadSecrets step further back (before the steps that creates the loadbalancer).

@plkokanov I did the same and still wasn't able to reproduce it ... Which means that even if there is an issue, this PR isn't likely to fix anything.
I still can't explain what I was able to observe on Monday though. I am quite sure I did the above steps while using the v1.8.0 gardenlet at this time. I could have made a mistake during the test, or perhaps the actual cause of the issue is a different one.
If you don't mind, let's keep the PR open for a while (I will convert it to draft), in case the issue is reproduced once again. If we don't see it within the next 1-2 weeks, I will close the PR.

If there are no objections and updates until tomorrow EOB (CEST) I would go ahead closing this issue


Is this project related to OpenJScad?

Its probably a trivial question, but it's my first time to "venture" into 3D modelling related stuff and because I usually do JavaScript, I'd like to have some synergy effects where possible.
I found blender to have a plugin to interface with OpenJScad, but now also stumbled upon node-occ.
I'm really new and lack all the fundamental basics that i should probably know about, hence my probably trivial question.
What is the background of this project? Why does it exist? How does it compare to alternatives :-)
If you could point me at some learning resources or give me a small overview so I have some orientation, that would be neat ;-)

I want to note that the difference I've spotted is that if a webapp uses a distant server to process scripts, then there is only a tiny amount of data should be sent/received back and forth. However, if user wants to implement some visual stuff to modify the code (such as dragging objects by mouse) there would be an extensive amount of data transmission needed which OpenJScad shouldn't need by definition. This, I think, the key difference between OpenJScad and node-occ.


What is a "segment" In Receive Segment Coalescing?

The documentation is vague. It says "When receiving data, the miniport driver, NDIS, and TCP/IP must all look at each segment's header information"
What is "Segment" when talking about packets? Does this mean the header of TCP/UDP, or just the header of IP layer?

Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: 54dbffcf-a1a5-4d77-df73-c4c8b9302f52
Version Independent ID: 92bdfa9d-715d-784a-a0bc-d9741301633e
Content: Overview of Receive Segment Coalescing - Windows drivers
Content Source: windows-driver-docs-pr/network/overview-of-receive-segment-coalescing.md
Product: windows-hardware
Technology: network
GitHub Login: @aviviano
Microsoft Alias: amvivian

Hi @bradhodge,
Regarding the line: "When receiving data, the miniport driver, NDIS, and TCP/IP must all look at each segment's header information." "Segment" here more precisely refers to a Protocol Data Unit (PDU). PDU refers to data packets as they are processed in the various layers of the OSI model. “Segment” typically refers to the transport layer PDU. In the quoted line, the “segment’s header information” refers to both the IP and TCP headers of the PDU. I've updated the line to: "When receiving data, the miniport driver, NDIS, and TCP/IP must all look at each PDU's header information."
Thanks very much for pointing this out.


Support for async chksum calculation

This PR adds support for async chksum calculation.
Support for various runtimes:

[ ] async-std
[ ] tokio
[ ] ?

Work in progress. Related issue chksum-rs/lib#1.

It seems that none of those runtimes support IsTerminal::is_terminal alternative.

Worth to follow:

async-rs/async-std#1073
tokio-rs/tokio#6407


BUG - Times series not showing

Cannot select time series.
See attachment for actual video
https://user-images.githubusercontent.com/85642641/130540966-e25282a1-f60e-411b-84d2-2dcd98b4d6e4.mp4

another 1
https://user-images.githubusercontent.com/85642641/130590301-1d3e1fa4-bf33-4449-95df-39f665dc36bf.mp4


ug update

Codecov Report
Base: 68.01% // Head: 68.01% // No change to project coverage :thumbsup:

Coverage data is based on head (a8673ce) compared to base (53da77d).
Patch has no changes to coverable lines.

Additional details and impacted files
@@            Coverage Diff            @@
##             master     #259   +/-   ##
=========================================
  Coverage     68.01%   68.01%
  Complexity      997      997
=========================================
  Files           153      153
  Lines          3320     3320
  Branches        453      453
=========================================
  Hits           2258     2258
  Misses          869      869
  Partials        193      193

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.


fileExtension  doesn't seem to work

We're trying to copy files from an SFTP folder to our blob storage. The source and sink are set up without a filename. I specified the .txt extension, but when running my pipeline, the blob storage shows the files being created as .csv. According to the documentation, they ought to use the extension supplied in the File Extension.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 2309a152-b9bc-7876-e010-b3bde6945d42
Version Independent ID: 751c7679-b01c-301d-8211-30f43ff50c54
Content: Delimited text format in Azure Data Factory
Content Source: articles/data-factory/format-delimited-text.md
Service: data-factory
GitHub Login: @linda33wj
Microsoft Alias: jingwang

@pes-kapacity  Thanks for the question! We are investigating and will update you shortly.

Hello @pes-kapacity and thank you for your inquiry.  I believe what you said is correct, however just to confirm my understanding, could you please share some screenshots or code?  Sharing this will aid in my attempt to reproduce and/or troubleshoot your issue.

Hi @MartinJaffer-MSFT. Thank you for picking it up :)
Here's the code (Mapping were removed to keep things brief):
{
"name": "CopyToBlob",
"type": "Copy",
"dependsOn": [
{
"activity": "Set LoadStart",
"dependencyConditions": [
"Succeeded"
]
}
],
"policy": {
"timeout": "7.00:00:00",
"retry": 0,
"retryIntervalInSeconds": 30,
"secureOutput": false,
"secureInput": false
},
"userProperties": [],
"typeProperties": {
"source": {
"type": "DelimitedTextSource",
"storeSettings": {
"type": "SftpReadSetting",
"recursive": false,
"wildcardFileName": "*.csv",
"modifiedDatetimeStart": {
"value": "@{variables('LastLoaded')}",
"type": "Expression"
}
},
"formatSettings": {
"type": "DelimitedTextReadSetting",
"skipLineCount": 1
}
},
"sink": {
"type": "DelimitedTextSink",
"storeSettings": {
"type": "AzureBlobStorageWriteSetting"
},
"formatSettings": {
"type": "DelimitedTextWriteSetting",
"quoteAllText": true,
"fileExtension": ".txt"
}
},
"enableStaging": false,
"translator": {
"type": "TabularTranslator",
"mappings": [

]
}
},
"inputs": [
{
"referenceName": "ChannelData_Customer",
"type": "DatasetReference"
}
],
"outputs": [
{
"referenceName": "ChannelData_CustomerData",
"type": "DatasetReference"
}
]
},

@pes-kapacity I have reproduced the issue.  In my blob storage, my test file had ending 'csv' when it should have been 'txt'.  Thank you for providing those details.  I will forward this to internal teams, and update here when I know more.

Hello @pes-kapacity .  This feature is a bit misleading in how it is presented.  Currently, the 'fileExtension' only takes effect when all three of the following conditions are true:

sinkFileName is empty
sourceFileName is empty
copyBehavior  is not PreserveHierarchy

The product group is aware of the issue.  At this moment, there is no timeline for a change.
Would you like me to work with you and find a suitable work-around?

Hi Martin,
Thanks for getting back to me. All those conditions are true in our case:
Here is the source:
<EMAIL_ADDRESS>Sink:
<EMAIL_ADDRESS>And this is the copy behavior:
<EMAIL_ADDRESS>So we meet all three conditions, but still see .csv files in the blob storage, rather than txt files.
Thanks 😊
Best,
Peter

From: MartinJaffer-MSFT<EMAIL_ADDRESS>Sent: Wednesday, July 31, 2019 3:54:34 AM
To: MicrosoftDocs/azure-docs<EMAIL_ADDRESS>Cc: Peter Schmitz<EMAIL_ADDRESS>Mention<EMAIL_ADDRESS>Subject: Re: [MicrosoftDocs/azure-docs] fileExtension doesn't seem to work (#35926)
Hello @pes-kapacityhttps://github.com/pes-kapacity . This feature is a bit misleading in how it is presented. Currently, the 'fileExtension' only takes effect when all three of the following conditions are true:

sinkFileName is empty
sourceFileName is empty
copyBehavior is not PreserveHierarchy

The product group is aware of the issue. At this moment, there is no timeline for a change.
Would you like me to work with you and find a suitable work-around?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/MicrosoftDocs/azure-docs/issues/35926?email_source=notifications&email_token=AMW6YTFL4K4E3H4BC46C56LQCDWFVA5CNFSM4IHDOHM2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3F2BWI#issuecomment-516661465, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AMW6YTDQTPZPGLTSLND4XODQCDWFVANCNFSM4IHDOHMQ.

Hi Peter, @pes-kapacity .  When you send an email to Github, the picture are stripped out, replaced by their names.  Could you please go to GitHub in your browser and insert your pictures that way?

Hi Martin,
I just tried editing the earlier post. The links to the images seem to work for me. I hope you can see them, too. If not, please let me know.
Thank you.
Best,
Peter

Thank you.  They are now links.  That works for me.  I will now escalate.

@pes-kapacity  Could you share with me a pipeline run ID and activity run ID?  You can get this by triggering the pipeline, and then going to the monitoring screen.  For this, do not use a 'debug' run.  Once I have these ID's I can send them to someone who can look into the back-end logs.

Hi Martin,
None of this has been published yet, as we're still developing it. I have another case related to the same branch that's basically preventing us from pushing this into a PR. So for the time being, the best I can do is run it in debug mode and provide you with the runID from there. That's what we have been doing with the other ticket, and that seems to give the team enough to go by.
Thanks.
Peter

From: MartinJaffer-MSFT<EMAIL_ADDRESS>Sent: Friday, August 2, 2019 8:59:15 PM
To: MicrosoftDocs/azure-docs<EMAIL_ADDRESS>Cc: Peter Schmitz<EMAIL_ADDRESS>Mention<EMAIL_ADDRESS>Subject: Re: [MicrosoftDocs/azure-docs] fileExtension doesn't seem to work (#35926)
@pes-kapacityhttps://github.com/pes-kapacity Could you share with me a pipeline run ID and activity run ID? You can get this by triggering the pipeline, and then going to the monitoring screen. For this, do not use a 'debug' run. Once I have these ID's I can send them to someone who can look into the back-end logs.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/MicrosoftDocs/azure-docs/issues/35926?email_source=notifications&email_token=AMW6YTBYGMGDIGAQUS5PIZLQCR7YHA5CNFSM4IHDOHM2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3OSPAI#issuecomment-517810049, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AMW6YTECOUCIMI34DHLNYNDQCR7YHANCNFSM4IHDOHMQ.

Okay.  I will take the debug mode ID's.  What is the other ticket, if I may ask?

Try this one (I just ran it):
Pipeline Run ID: 649ca0be-d7d4-4ca8-9f52-ffa34e37922d
The task called "Copy Data" holds the Delimiter source object. The flow is actually working fine as is. The reason we tried saving them to text files is because a colleague mentioned the other issue could be solved by sourcing the file as a text-file rather than a delimited file. However, I tried simply renaming the file(s) on the FTP Server to .txt instead of.csv, and that did absolutely nothing.
The other issue is related to another pipeline within the same branch. Like the pipeline you just received the RunID for, we source delimited files from an SFTP server, write them t our blob storage (for archiving purposes), and then source them from the BLOB. At least, that was the plan. We have run into issues with these other files. The source files are delimited with a "~" symbol. However, we have experienced numerous cases where the files won't process, and Data Factory specifies an error that the file contains more columns than it expects. We thought we narrowed it down to some of the text entries containing a semi-colon, but I've now seen it fail on totally normal files, too.
I managed to sometimes get individual files processed correctly if I played with the advanced settings (degree of parallelism and number of DIUs), but we have so far not managed to get it to work on the entire folder. Seeing these files are crucial to our customer's sales processes, we really need to have figured out what's going wrong. For the time being, we will rely on an on-prem SSIS package to source them, but ideally, we'd be able to simply process them within Azure, so we can slowly replace the on-prem solutions with cloud-based ones.
There's a thread on it here:
https://social.msdn.microsoft.com/Forums/en-US/be063cd5-b02a-4f9f-91a7-41bbeacef341/adf-v2-issue-with-delimiter-when-dealing-with-a-connectionthrottled-sftp-server-and-big-files?forum=AzureDataFactory
And we have been in touch with a few support people in (presumably) China. We also were granted a free ticket, but neither my direct customer nor myself have the necessary permissions to see the page required to file a ticket (and the 1 person who could grant us the permissions is on vacation), so unfortunately I do not have an exact ticket number for you.
Thanks!
Best regards,
Peter

From: MartinJaffer-MSFT<EMAIL_ADDRESS>Sent: Friday, August 2, 2019 9:07:36 PM
To: MicrosoftDocs/azure-docs<EMAIL_ADDRESS>Cc: Peter Schmitz<EMAIL_ADDRESS>Mention<EMAIL_ADDRESS>Subject: Re: [MicrosoftDocs/azure-docs] fileExtension doesn't seem to work (#35926)
Okay. I will take the debug mode ID's. What is the other ticket, if I may ask?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/MicrosoftDocs/azure-docs/issues/35926?email_source=notifications&email_token=AMW6YTERVHZ7VJU5M5YQON3QCSAXRA5CNFSM4IHDOHM2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3OTBLI#issuecomment-517812397, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AMW6YTFR2UYITOEWRF2U27DQCSAXRANCNFSM4IHDOHMQ.

Heh.
The pipeline RunID I just provided you with also threw the error. That's a first for this set of files. So far, I've only seen it on the sales files.
That's a bummer. I thought we had 2 out of the 3 sets of files nailed. I'll have to investigate some time next week what happened.
/Peter

From: Peter Schmitz<EMAIL_ADDRESS>Sent: Friday, August 2, 2019 9:20:19 PM
To: MicrosoftDocs/azure-docs<EMAIL_ADDRESS>MicrosoftDocs/azure-docs<EMAIL_ADDRESS>Cc: Mention<EMAIL_ADDRESS>Subject: Re: [MicrosoftDocs/azure-docs] fileExtension doesn't seem to work (#35926)
Try this one (I just ran it):
Pipeline Run ID: 649ca0be-d7d4-4ca8-9f52-ffa34e37922d
The task called "Copy Data" holds the Delimiter source object. The flow is actually working fine as is. The reason we tried saving them to text files is because a colleague mentioned the other issue could be solved by sourcing the file as a text-file rather than a delimited file. However, I tried simply renaming the file(s) on the FTP Server to .txt instead of.csv, and that did absolutely nothing.
The other issue is related to another pipeline within the same branch. Like the pipeline you just received the RunID for, we source delimited files from an SFTP server, write them t our blob storage (for archiving purposes), and then source them from the BLOB. At least, that was the plan. We have run into issues with these other files. The source files are delimited with a "~" symbol. However, we have experienced numerous cases where the files won't process, and Data Factory specifies an error that the file contains more columns than it expects. We thought we narrowed it down to some of the text entries containing a semi-colon, but I've now seen it fail on totally normal files, too.
I managed to sometimes get individual files processed correctly if I played with the advanced settings (degree of parallelism and number of DIUs), but we have so far not managed to get it to work on the entire folder. Seeing these files are crucial to our customer's sales processes, we really need to have figured out what's going wrong. For the time being, we will rely on an on-prem SSIS package to source them, but ideally, we'd be able to simply process them within Azure, so we can slowly replace the on-prem solutions with cloud-based ones.
There's a thread on it here:
https://social.msdn.microsoft.com/Forums/en-US/be063cd5-b02a-4f9f-91a7-41bbeacef341/adf-v2-issue-with-delimiter-when-dealing-with-a-connectionthrottled-sftp-server-and-big-files?forum=AzureDataFactory
And we have been in touch with a few support people in (presumably) China. We also were granted a free ticket, but neither my direct customer nor myself have the necessary permissions to see the page required to file a ticket (and the 1 person who could grant us the permissions is on vacation), so unfortunately I do not have an exact ticket number for you.
Thanks!
Best regards,
Peter

From: MartinJaffer-MSFT<EMAIL_ADDRESS>Sent: Friday, August 2, 2019 9:07:36 PM
To: MicrosoftDocs/azure-docs<EMAIL_ADDRESS>Cc: Peter Schmitz<EMAIL_ADDRESS>Mention<EMAIL_ADDRESS>Subject: Re: [MicrosoftDocs/azure-docs] fileExtension doesn't seem to work (#35926)
Okay. I will take the debug mode ID's. What is the other ticket, if I may ask?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/MicrosoftDocs/azure-docs/issues/35926?email_source=notifications&email_token=AMW6YTERVHZ7VJU5M5YQON3QCSAXRA5CNFSM4IHDOHM2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3OTBLI#issuecomment-517812397, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AMW6YTFR2UYITOEWRF2U27DQCSAXRANCNFSM4IHDOHMQ.

Ahh I do have an idea for you now.  I don't think they meant changing the file extension, but instead using a different connector.  I can't imagine the file suffix imacting anythign.  You see, go to look at the JSON definition of your activitiy and datasets.  You will find "Type":"DelimitedText".  I believe they meant to try an older module ,"TextFormat".
To see the two different dataset types in the UI... umm, nevermind on that.  Looks like things were changed since 2 days ago.  I'll share with you some JSON instead.
Here is a dataset I used while attempting to repro and find work-arounds for you.
{     "name": "RebexSourceNotDelimited",     "properties": {         "linkedServiceName": {             "referenceName": "Rebexlocal sftp",             "type": "LinkedServiceReference"         },         "annotations": [],         "type": "FileShare",         "typeProperties": {             "format": {                 "type": "TextFormat",                 "columnDelimiter": ",",                 "rowDelimiter": "",                 "treatEmptyAsNull": true,                 "skipLineCount": 0,                 "firstRowAsHeader": true             },             "fileName": "*.csv"         }     } }
This is an example of an older interface, but I believe is what was meant.

Oh, I found how to still get at the TextFormat in UI.

From here, do not select DelimitedText.  Instead choose Json or Avro.

Now, under File format, change from Json/Avro to 'Text Format'.  I was thrown off because previously, 'Binary' also directed here.

That could very well be. I actually looked for a non-delimited text source, but didn't find one. It could very well be my colleague tried that before her vacation, and by the time I tried reproducing it, the source was gone. I highly enjoy the constant changes and improvements, but at times it does make yhou question your own sanity if something you saw the day before suddenly is behaving or looking slightly different 😃
I think my colleague is back next week. I'll have a word with her then, but still hope that we can figure out why we keep running into these errors, and get it resolved in the components.
Thanks :)
/Peter

From: MartinJaffer-MSFT<EMAIL_ADDRESS>Sent: Friday, August 2, 2019 10:03:25 PM
To: MicrosoftDocs/azure-docs<EMAIL_ADDRESS>Cc: Peter Schmitz<EMAIL_ADDRESS>Mention<EMAIL_ADDRESS>Subject: Re: [MicrosoftDocs/azure-docs] fileExtension doesn't seem to work (#35926)
Oh, I found how to still get at the TextFormat in UI.
[image]https://user-images.githubusercontent.com/46612670/62395518-5a9a7480-b525-11e9-9f0a-b3c4ffdb5cfd.png
From here, do not select DelimitedText. Instead choose Json or Avro.
[image]https://user-images.githubusercontent.com/46612670/62395599-99c8c580-b525-11e9-8792-e12b38da2b57.png
Now, under File format, change from Json/Avro to 'Text Format'. I was thrown off because previously, 'Binary' also directed here.
[image]https://user-images.githubusercontent.com/46612670/62395683-db597080-b525-11e9-9080-1a822d313510.png
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/MicrosoftDocs/azure-docs/issues/35926?email_source=notifications&email_token=AMW6YTEPRTQTLGSSRYPYKE3QCSHI3A5CNFSM4IHDOHM2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3OWYOQ#issuecomment-517827642, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AMW6YTG3MUPXJG57CXMVFN3QCSHI3ANCNFSM4IHDOHMQ.

Well, you could always just do a binary copy.  That moves it as-is.  No changes whatsoever.

If you don't mind, I'll close this issue.  If need be, we can reopen again later.

Hi Martin,
Is the issue actually resolved, though? My main reason for opening it to begin with was that the component, specifically the file extension, does not work the way the documentation says it would. It won't help me in this case, but in my opinion, it doesn't work as intended (or the documentation should be updated).
Thanks,
Peter

From: MartinJaffer-MSFT<EMAIL_ADDRESS>Sent: Friday, August 2, 2019 11:10:04 PM
To: MicrosoftDocs/azure-docs<EMAIL_ADDRESS>Cc: Peter Schmitz<EMAIL_ADDRESS>Mention<EMAIL_ADDRESS>Subject: Re: [MicrosoftDocs/azure-docs] fileExtension doesn't seem to work (#35926)
If you don't mind, I'll close this issue. If need be, we can reopen again later.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/MicrosoftDocs/azure-docs/issues/35926?email_source=notifications&email_token=AMW6YTAIKKEWEIXGULETOIDQCSPCZA5CNFSM4IHDOHM2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3O3BZQ#issuecomment-517845222, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AMW6YTD33YTVF5XHDAKK37TQCSPCZANCNFSM4IHDOHMQ.

I agree.  The feature is misleading.  I cited your example, and how you appeared to meet all three conditions.  Here is the response I got:

in your scenario (copy from folder to folder), if you specify the copy behavior is None, we will use "PreserveHierarchy" as the default behavior. So the file extension doesn't work. We are preparing to improve the use experience in this scenario and well handle the file extension. Will let you know when we finish this improvement.

Cool, thank you. That bit with "When None is selected, PreserveHierarchy becomes default" is definitely confusing, lol.

From: MartinJaffer-MSFT<EMAIL_ADDRESS>Sent: Monday, August 5, 2019 6:55:36 PM
To: MicrosoftDocs/azure-docs<EMAIL_ADDRESS>Cc: Peter Schmitz<EMAIL_ADDRESS>Mention<EMAIL_ADDRESS>Subject: Re: [MicrosoftDocs/azure-docs] fileExtension doesn't seem to work (#35926)
I agree. The feature is misleading. I cited your example, and how you appeared to meet all three conditions. Here is the response I got:
in your scenario (copy from folder to folder), if you specify the copy behavior is None, we will use "PreserveHierarchy" as the default behavior. So the file extension doesn't work. We are preparing to improve the use experience in this scenario and well handle the file extension. Will let you know when we finish this improvement.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/MicrosoftDocs/azure-docs/issues/35926?email_source=notifications&email_token=AMW6YTCJ2GKVMAXQUFTCJE3QDBLQRA5CNFSM4IHDOHM2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3SNMVA#issuecomment-518313556, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AMW6YTC2IGTZ7B7DSG55SNDQDBLQRANCNFSM4IHDOHMQ.

Totally.

Did the non-delimited text thing help?  I totally agree, changes cause me to question everything too.


COMP: libjpeg-turbo: Make linkage private to object libraries

Addresses #5021.

PR Checklist

[ ] No API changes were made (or the changes have been approved)
[ ] No major design changes were made (or the changes have been approved)
[ ] Added test (or behavior not changed)
[ ] Updated API documentation (or API not changed)
[ ] Added license to new files (if any)
[ ] Added Python wrapping to new files (if any) as described in ITK Software Guide Section 9.5
[ ] Added ITK examples for all new major features (if any)

Refer to the ITK Software Guide for
further development details if necessary.

Unfortunately, this does not work:
Build started at 12:20...
1>------ Build started: Project: ZERO_CHECK, Configuration: Debug x64 ------
2>------ Skipped Build: Project: RUN_TESTS, Configuration: Debug x64 ------
2>Project not selected to build for this solution configuration
3>------ Skipped Build: Project: NightlyMemoryCheck, Configuration: Debug x64 ------
3>Project not selected to build for this solution configuration
4>------ Skipped Build: Project: Nightly, Configuration: Debug x64 ------
4>Project not selected to build for this solution configuration
5>------ Build started: Project: FpfhTestDriver, Configuration: Debug x64 ------
6>------ Build started: Project: FpfhHeaderTestClean, Configuration: Debug x64 ------
7>------ Skipped Build: Project: Experimental, Configuration: Debug x64 ------
7>Project not selected to build for this solution configuration
8>------ Skipped Build: Project: Continuous, Configuration: Debug x64 ------
8>Project not selected to build for this solution configuration
9>------ Skipped Build: Project: ITKHeaderTests, Configuration: Debug x64 ------
9>Project not selected to build for this solution configuration
5>itkjpeg-6.0.lib(jdmaster.obj) : error LNK2019: unresolved external symbol itk_j12init_d_main_controller referenced in function master_selection
5>itkjpeg-6.0.lib(jdmaster.obj) : error LNK2019: unresolved external symbol itk_j12init_d_coef_controller referenced in function master_selection
5>itkjpeg-6.0.lib(jdmaster.obj) : error LNK2019: unresolved external symbol itk_j12init_d_post_controller referenced in function master_selection
5>itkjpeg-6.0.lib(jdmaster.obj) : error LNK2019: unresolved external symbol itk_j12init_inverse_dct referenced in function master_selection
5>itkjpeg-6.0.lib(jdmaster.obj) : error LNK2019: unresolved external symbol itk_j12init_upsampler referenced in function master_selection
5>itkjpeg-6.0.lib(jdmaster.obj) : error LNK2019: unresolved external symbol itk_j12init_color_deconverter referenced in function master_selection
5>itkjpeg-6.0.lib(jdmaster.obj) : error LNK2019: unresolved external symbol itk_j12init_1pass_quantizer referenced in function master_selection
5>itkjpeg-6.0.lib(jdmaster.obj) : error LNK2019: unresolved external symbol itk_j12init_2pass_quantizer referenced in function master_selection
5>itkjpeg-6.0.lib(jdmaster.obj) : error LNK2019: unresolved external symbol itk_j12init_merged_upsampler referenced in function master_selection
5>itkjpeg-6.0.lib(jdmaster.obj) : error LNK2019: unresolved external symbol itk_j16init_d_main_controller referenced in function master_selection
5>itkjpeg-6.0.lib(jdmaster.obj) : error LNK2019: unresolved external symbol itk_j16init_d_post_controller referenced in function master_selection
5>itkjpeg-6.0.lib(jdmaster.obj) : error LNK2019: unresolved external symbol itk_j16init_upsampler referenced in function master_selection
5>itkjpeg-6.0.lib(jdmaster.obj) : error LNK2019: unresolved external symbol itk_j16init_color_deconverter referenced in function master_selection
5>itkjpeg-6.0.lib(jdmaster.obj) : error LNK2019: unresolved external symbol itk_j12init_d_diff_controller referenced in function master_selection
5>itkjpeg-6.0.lib(jdmaster.obj) : error LNK2019: unresolved external symbol itk_j16init_d_diff_controller referenced in function master_selection
5>itkjpeg-6.0.lib(jdmaster.obj) : error LNK2019: unresolved external symbol itk_j12init_lossless_decompressor referenced in function master_selection
5>itkjpeg-6.0.lib(jdmaster.obj) : error LNK2019: unresolved external symbol itk_j16init_lossless_decompressor referenced in function master_selection
5>itkjpeg-6.0.lib(jcinit.obj) : error LNK2019: unresolved external symbol itk_j12init_c_main_controller referenced in function itk_jinit_compress_master
5>itkjpeg-6.0.lib(jcinit.obj) : error LNK2019: unresolved external symbol itk_j12init_c_prep_controller referenced in function itk_jinit_compress_master
5>itkjpeg-6.0.lib(jcinit.obj) : error LNK2019: unresolved external symbol itk_j12init_c_coef_controller referenced in function itk_jinit_compress_master
5>itkjpeg-6.0.lib(jcinit.obj) : error LNK2019: unresolved external symbol itk_j12init_color_converter referenced in function itk_jinit_compress_master
5>itkjpeg-6.0.lib(jcinit.obj) : error LNK2019: unresolved external symbol itk_j12init_downsampler referenced in function itk_jinit_compress_master
5>itkjpeg-6.0.lib(jcinit.obj) : error LNK2019: unresolved external symbol itk_j12init_forward_dct referenced in function itk_jinit_compress_master
5>itkjpeg-6.0.lib(jcinit.obj) : error LNK2019: unresolved external symbol itk_j16init_c_main_controller referenced in function itk_jinit_compress_master
5>itkjpeg-6.0.lib(jcinit.obj) : error LNK2019: unresolved external symbol itk_j16init_c_prep_controller referenced in function itk_jinit_compress_master
5>itkjpeg-6.0.lib(jcinit.obj) : error LNK2019: unresolved external symbol itk_j16init_color_converter referenced in function itk_jinit_compress_master
5>itkjpeg-6.0.lib(jcinit.obj) : error LNK2019: unresolved external symbol itk_j16init_downsampler referenced in function itk_jinit_compress_master
5>itkjpeg-6.0.lib(jcinit.obj) : error LNK2019: unresolved external symbol itk_j12init_c_diff_controller referenced in function itk_jinit_compress_master
5>itkjpeg-6.0.lib(jcinit.obj) : error LNK2019: unresolved external symbol itk_j16init_c_diff_controller referenced in function itk_jinit_compress_master
5>itkjpeg-6.0.lib(jcinit.obj) : error LNK2019: unresolved external symbol itk_j12init_lossless_compressor referenced in function itk_jinit_compress_master
5>itkjpeg-6.0.lib(jcinit.obj) : error LNK2019: unresolved external symbol itk_j16init_lossless_compressor referenced in function itk_jinit_compress_master
5>C:\Misc\ITK-patches2-vs22\bin\Debug\FpfhTestDriver.exe : fatal error LNK1120: 31 unresolved externals
5>Done building project "FpfhTestDriver.vcxproj" -- FAILED.
10>------ Skipped Build: Project: INSTALL, Configuration: Debug x64 ------
10>Project not selected to build for this solution configuration
========== Build: 2 succeeded, 1 failed, 4 up-to-date, 7 skipped ==========
========== Build completed at 12:20 and took 01.027 seconds ==========

pixi builds seem to be timing out. Merging.


Core -> Asset manager -> Image Optimization:: Smart image crop & scale

As a PO, I would like to have cropping and scaling features with AI. Example and doc could be found here: https://elixirforum.com/t/image-an-image-processing-library-based-upon-vix/47568/104?u=leandrocp

https://elixirforum.com/t/image-an-image-processing-library-based-upon-vix/47568/102

https://github.com/BeaconCMS/beacon/pull/270


External Authentication can create new headers, but not overwrite existing headers

Describe the bug
Allowed headers from the external authentication response are copied into the upstream request if and only if that header does not already exist.
To Reproduce
Sample configuration:
apiVersion: ambassador/v0
kind:  AuthService
name:  authentication
auth_service: "hostname:8001"
path_prefix: "/authenticate"
allowed_headers:
- "Set-cookie"
- "Authorization"

I sent in a request that had the value 'jk' in the Authorization header. Here are the envoy logs for the response from the external authentication and the resulting upstream request with the original Authorization header.
[2018-05-30 18:30:09.281][21][debug][router] source/common/router/router.cc:573] [C0][S16497426937251309152] upstream headers complete: end_stream=false
[2018-05-30 18:30:09.281][21][debug][http] source/common/http/async_client_impl.cc:85] async http request response headers (end_stream=false):
[2018-05-30 18:30:09.281][21][debug][http] source/common/http/async_client_impl.cc:88]   'date':'Wed, 30 May 2018 18:30:09 GMT'
[2018-05-30 18:30:09.281][21][debug][http] source/common/http/async_client_impl.cc:88]   'access-control-allow-origin':'*'
[2018-05-30 18:30:09.282][21][debug][http] source/common/http/async_client_impl.cc:88]   'content-encoding':'gzip'
[2018-05-30 18:30:09.282][21][debug][http] source/common/http/async_client_impl.cc:88]   'cache-control':'no-store, no-cache, must-revalidate, proxy-revalidate'
[2018-05-30 18:30:09.282][21][debug][http] source/common/http/async_client_impl.cc:88]   'pragma':'no-cache'
[2018-05-30 18:30:09.282][21][debug][http] source/common/http/async_client_impl.cc:88]   'expires':'0'
[2018-05-30 18:30:09.282][21][debug][http] source/common/http/async_client_impl.cc:88]   'content-type':'application/json'
[2018-05-30 18:30:09.282][21][debug][http] source/common/http/async_client_impl.cc:88]   'authorization':'Bearer eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICJ2dVdPcGJkQW5ZSzl0OGx4cXRYSHBMOGVLeDlJQzhDRnc1RUJVdXpXNm40In0.eyJqdGkiOiJiM2NlYzE1Zi1hNmYxLTQ1MTgtOTNkOS05ODY5ZmI1ZDcxMDIiLCJleHAiOjE1Mjc3MDg1OTIsIm5iZiI6MCwiaWF0IjoxNTI3NzA0OTkyLCJpc3MiOiJodHRwOi8va2V5Y2xvYWs6ODA4MC9hdXRoL3JlYWxtcy9tYXN0ZXIiLCJhdWQiOiJ1c2VyLXNlcnZpY2UiLCJzdWIiOiIxY2I0NjFhNy0xMWE2LTQwZWMtYmRiYi1iZGNmNTdjNDQ1NDMiLCJ0eXAiOiJCZWFyZXIiLCJhenAiOiJ1c2VyLXNlcnZpY2UiLCJub25jZSI6ImE1cURzN0YxRUoyd3lxMmlxckhKc0J2bW44VzZhT1lkcUpwcFl4TXVWZTQiLCJhdXRoX3RpbWUiOjE1Mjc3MDQ5MjQsInNlc3Npb25fc3RhdGUiOiI2YTU5ZTg3YS0zMjBiLTQ1Y2EtYWQyYS0zMGQ1OTZiNDgwMzQiLCJhY3IiOiIwIiwiYWxsb3dlZC1vcmlnaW5zIjpbXSwicmVhbG1fYWNjZXNzIjp7InJvbGVzIjpbImNyZWF0ZS1yZWFsbSIsImFkbWluIiwidW1hX2F1dGhvcml6YXRpb24iXX0sInJlc291cmNlX2FjY2VzcyI6eyJtYXN0ZXItcmVhbG0iOnsicm9sZXMiOlsidmlldy1pZGVudGl0eS1wcm92aWRlcnMiLCJ2aWV3LXJlYWxtIiwibWFuYWdlLWlkZW50aXR5LXByb3ZpZGVycyIsImltcGVyc29uYXRpb24iLCJjcmVhdGUtY2xpZW50IiwibWFuYWdlLXVzZXJzIiwicXVlcnktcmVhbG1zIiwidmlldy1hdXRob3JpemF0aW9uIiwicXVlcnktY2xpZW50cyIsInF1ZXJ5LXVzZXJzIiwibWFuYWdlLWV2ZW50cyIsIm1hbmFnZS1yZWFsbSIsInZpZXctZXZlbnRzIiwidmlldy11c2VycyIsInZpZXctY2xpZW50cyIsIm1hbmFnZS1hdXRob3JpemF0aW9uIiwibWFuYWdlLWNsaWVudHMiLCJxdWVyeS1ncm91cHMiXX0sImFjY291bnQiOnsicm9sZXMiOlsibWFuYWdlLWFjY291bnQiLCJtYW5hZ2UtYWNjb3VudC1saW5rcyIsInZpZXctcHJvZmlsZSJdfX0sInByZWZlcnJlZF91c2VybmFtZSI6ImFkbWluIn0.YjFEkTyFMKdN4xHhSOUP-2E2wsD5GTE4Dz5XsROkTX89K2HHUacM6-V6LxKhxOsqnk40mh2y0rjYdDcTrsY-F_ZY_uEOPitIuiLnuwzMGZkYAxaNatw-Vw0QR3jTzdOJhG9NkLLdjZbunko-TXAjh6Klmfn3qrzHYJODCdcFHcyJcUf1dN8SQti_XEUadxAEGGU5lmDL7ihmrJVAOMpd-FdegNe2f4MO-17yzrbwhwTswAmH27kSzlLibxE6YaScU7E35Z8FBhqEbCTr_oX9Z0MheO1XAxxg2D8N_yCu52BnlzLQj0uFV7Ct0O30EBciJI00edHcYvWBp5G_4wKrZQ'
[2018-05-30 18:30:09.282][21][debug][http] source/common/http/async_client_impl.cc:88]   'transfer-encoding':'chunked'
[2018-05-30 18:30:09.282][21][debug][http] source/common/http/async_client_impl.cc:88]   'server':'Jetty(9.4.z-SNAPSHOT)'
[2018-05-30 18:30:09.282][21][debug][http] source/common/http/async_client_impl.cc:88]   ':status':'200'
[2018-05-30 18:30:09.282][21][debug][http] source/common/http/async_client_impl.cc:88]   'x-envoy-upstream-service-time':'3'
[2018-05-30 18:30:09.283][21][debug][client] source/common/http/codec_client.cc:81] [C107] response complete
[2018-05-30 18:30:09.283][21][debug][filter] source/common/http/filter/extauth.cc:223] [C103][S5397472963650963421] ExtAuth accepting request
[2018-05-30 18:30:09.283][21][debug][router] source/common/router/router.cc:248] [C103][S5397472963650963421] cluster 'cluster_system' match for URL '/api/system/systems'
[2018-05-30 18:30:09.284][21][debug][router] source/common/router/router.cc:293] [C103][S5397472963650963421]   ':authority':'localhost:9999'
[2018-05-30 18:30:09.284][21][debug][router] source/common/router/router.cc:293] [C103][S5397472963650963421]   'accept':'application/json'
[2018-05-30 18:30:09.284][21][debug][router] source/common/router/router.cc:293] [C103][S5397472963650963421]   'authorization':'jk'

If the incoming request does not have an Authorization header at all, then envoy copies the Authorization header provided by the External Authorization service as expected.
Expected behavior
Allowed Headers returned by the External Authorization service should replace existing headers in the upstream request.
Versions (please complete the following information):

Ambassador: 0.33.1

Additional context
Here is an example use case: an incoming request has an expired JWT in the Authorization header. My external authorization service has the ability to refresh that token but cannot do so because it cannot provide new content for a header that already exists in the request.

Confirmed, sadly enough -- if you're using some arbitrary X- header it works fine, but if you're trying to overwrite Authorization, it doesn't. This is almost certainly a bug in the Envoy extauth filter itself related to Envoy's fastpath vs slowpath header handling. As such it'll require a filter change and a new ambassador-envoy build (neither of which require an Envoy release, FTR).

The problem is identified, and fixed it in https://github.com/datawire/envoy/commit/d1579ff3bf9a3cd854c7d28ba7c39c4aa3f83b89 and https://github.com/datawire/envoy/commit/6a44698122d7c992beab89915d5caf7b6f911fe8. We need do a full regression test and ship the fix.

Fixed in 0.34.2.


feat: add outDir support to qwikInsights

Overview

I'm using qwik-nx which makes use of outDir for both client/server outputs. qwikInsights doesn't yet support it so the file ends up in a different location. Also, the manifest is assumed to exist within dist which isn't true with a qwik-nx project.
What is it?

[x] Feature / enhancement
[ ] Bug
[ ] Docs / tests / types / typos

Description
Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.
Adds support for outDir, which qwikVite already supports.
Use cases and why

To customise the outDir, which is useful in projects that are generated by qwik-nx where you can have multiple apps.

Checklist:

[x] My code follows the developer guidelines of this project
[x] I have performed a self-review of my own code
[ ] I have made corresponding changes to the documentation
[ ] Added new tests to cover the fix / functionality

Patch package:
<EMAIL_ADDRESS>diff --git<EMAIL_ADDRESS><EMAIL_ADDRESS>index 6cfa70b..84024fa 100644
---<EMAIL_ADDRESS>+++<EMAIL_ADDRESS>@@ -2,4 +2,5 @@ import { PluginOption } from 'vite';
 export declare function qwikInsights(qwikInsightsOpts: {
     publicApiKey: string;
     baseUrl?: string;
+    outDir?: string;
 }): Promise<PluginOption>;
diff --git<EMAIL_ADDRESS><EMAIL_ADDRESS>index eb72555..cbf7909 100644
---<EMAIL_ADDRESS>+++<EMAIL_ADDRESS>@@ -8,9 +8,8 @@ const logWarn = (message) => {
     console.warn('\x1b[33m%s\x1b[0m', `\n\nQWIK WARN: ${message}\n`);
 };
 async function qwikInsights(qwikInsightsOpts) {
-    const { publicApiKey, baseUrl = 'https://qwik-insights.builder.io' } = qwikInsightsOpts;
+    const { publicApiKey, baseUrl = 'https://qwik-insights.builder.io', outDir = 'dist' } = qwikInsightsOpts;
     let isProd = false;
-    const outDir = 'dist';
     const vitePlugin = {
         name: 'vite-plugin-qwik-insights',
         enforce: 'pre',

Thanks for the feedback, I will work on it. I will add a outdir prop to qwikInsights plugin. Is a good enough for nx?


Deleting card.js from the filebrowser doesn't update the canvas.

After deleting the file the canvas still renders the Card component.
Inserting new components/divs from the insertmenu to the parent App still keeps the Card on the canvas. The insertmenu correctly doesn't show the Card, because it was deleted.
How it should work: there is an error on the canvas when using a non-existing component.
After reloading the correct error shows up.

I've just reproduced this on pizza. (Both removing the import { Card } from /src/card.jsand removing theexport const Card = ... ` now are working correctly, but deleting card.js via the file browser still causes this problem)

Fixed by https://github.com/concrete-utopia/utopia/pull/1335


Changed egress to ingress

There is also the description field on SecurityGroupIngress which says: Description of this egress rule.. But i was unable to find where this came from.
https://www.pulumi.com/docs/reference/pkg/aws/ec2/securitygroup

@alexbowers Thanks for noticing the typo and submitting a PR! These come from the upstream docs and should be fixed there: https://github.com/pulumi/terraform-provider-aws/blob/upstream-v3.47.0/website/docs/r/security_group.html.markdown -- you could submit a PR against our branch or I would suggest submitting against the upstream provider (https://github.com/hashicorp/terraform-provider-aws) and we can pick up the change from there.


Adding new FrontDoor reliability recommendation

Added "Use the same domain name on Front Door and your origin"
Removed (temporarily) "Use managed TLS certificates". It will be added back after some technical issues are sorted out

#sign-off


test: change in hasura-auth

Codecov Report

:exclamation: No coverage uploaded for pull request base (main@31d3d09). Click here to learn what that means.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main       #2   +/-   ##
=======================================
  Coverage        ?   94.75%
=======================================
  Files           ?       48
  Lines           ?     6907
  Branches        ?      445
=======================================
  Hits            ?     6545
  Misses          ?      362
  Partials        ?        0

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more


[Introspection] optional ID

@do4gr has a theory how this could happen - did not see this yet though.

Still did not find that after looking through all of them


Explain how to prepare a Go 1.5 development environment

Other minor documentation fixes

Please sign your commits following these rules:
https://github.com/docker/docker/blob/master/CONTRIBUTING.md#sign-your-work
The easiest way to do this is to amend the last commit:
$ git clone -b "master" git@github.com:bookerzzz/machine.git somewhere
$ cd somewhere
$ git commit --amend -s --no-edit
$ git push -f

Ammending updates the existing PR. You DO NOT need to open a new one.

Hm, @dmp42 has some changes in the pipe for this doc as well which I will probably merge first, but I'm interested, so stay tuned.

@gdm85 do you think you can rebase now the other PRs have been merged? Thanks a lot.

@dmp42 done.
Please note that I am omitting the link to Go 1.5 installation on purpose (I don't think it fits the scope of this documentation, but it's arguable) and then providing an example about how to work on sources directly under a GOPATH. I am not sure if this is a best practice or not, but I couldn't figure out a better way to reuse the project-provided dependencies.

Thanks a lot @gdm85 !
I added a couple of remarks.
Also, we should still separate the native golang approach versus the docker only approach (people going with the latter don't need golang at all).

@dmp42 I am still learning markdown, apparently :)
As per the approaches split, do you think it should be made visible in a different way?

I've removed also the extra GOPATH, tested, it works

We could probably add the default simplest way of getting ready to code on machine which is @dgageot setup, this is what we want to achieve with this guide, make it easy to contribute.
And add an optional setup, where you don't share you GOPATH with other projects ( as @gdm85 suggested ).
As my personal experience, I did them both, and there is no doubt the former is easier.

@dmp42 IMHO this could be merged "as is", as it brings value right now. We should decide of the easiest way of getting up to speed on the project in another dedicated PR.

Please sign your commits following these rules:
https://github.com/docker/docker/blob/master/CONTRIBUTING.md#sign-your-work
The easiest way to do this is to amend the last commit:
$ git clone -b "master" git@github.com:bookerzzz/machine.git somewhere
$ cd somewhere
$ git rebase -i HEAD~2
editor opens
change each 'pick' to 'edit'
save the file and quit
$ git commit --amend -s --no-edit
$ git rebase --continue # and repeat the amend for each commit
$ git push -f

Ammending updates the existing PR. You DO NOT need to open a new one.

@jeanlaurent my personal opinion is that GOPATH is something that should be replaced by a command-line option (as it works in proven compilers like gcc/g++ with -I), but that is off-topic here :)
What I wanted to contribute here is a way for even an occasional contributor to do the right thing; I find myself contributing after weeks or a month and not updating the packages in GOPATH is dangerous in such cases.
With my latest commit 8ac92c2 I simplified the process even further by making use of go get; I didn't squash it to leave the option to you guys to either use it or drop it.

@dgageot ops, added the missing go get now :)

thx @gdm85
@dmp42 ready to review & merge.

LGTM
@jeanlaurent you can hit that merge button at will.

Should GOPATH really point to the Docker Machine repo's location on disk?  I'd generally be inclined towards recommending export GOPATH="$(pwd)/vendor:$GOPATH" or, better yet, export GO15VENDOREXPERIMENT=1.

GO15VENDOREXPERIMENT=1 will not help (will just make golang resolve vendor but won't add the current directory to GOPATH)
Exporting GOPATH="$(pwd)/vendor:$GOPATH" is usually the right approach in a proper golang setup where you work on multiple different stuff - but it might also create difficulties for newcomers wrt package resolution, and is not needed per-se if you just want to hack on machine.
Either way, a follow-up PR can certainly enhance this if this is a problem.

@nathanleclaire now that you mention it, I notice that my wording is misleading as I wrote to use Go 1.5 vendoring although this project doesn't use it yet; sorry for that.
In this PR I saw the approach more from the perspective of allowing contributions without having to fetch sub-packages in a separate GOPATH, thus as easy and self-contained as possible.

@gdm85 No problem, looks like a good start for now, and I'll revisit and PR any nitpicks I have.


Add docs for volume plugin ls/get

ping @thaJeztah

Thanks! 1 question, otherwise LGTM
I guess this also needs changes to https://github.com/docker/go-plugins-helpers/tree/master/volume ?

Updated

great! LGTM
ping @moxiegirl for review

lgtm @cpuguy83 thanks!


why I cannot save the DataFrame to pickle?

However, if your objects are not strings, then it might not work at all (and would be highly inefficient in any event).

http://bugs.python.org/issue11564


Remove unused imports

Might be time to finally get around to adding checkstyle...

Yup either that or PMD


[bug] sar_force_fov doesn't work after quickload in mel

In PS:Mel, when you load a save, fov goes back to 90 instead of wanted value with sar_force_fov.
Route credit: Extension

lol nekz' code assumes there will never be more than 16 ticks of demo delay. alas, mel big


Implement the ReplicateBlobRequest handling in AmbryServer.

Implement the ReplicateBlobRequest handling in AmbryServer.

Codecov Report
Base: 73.03% // Head: 14.98% // Decreases project coverage by -58.05% :warning:

Coverage data is based on head (cdbc60c) compared to base (5251f63).
Patch coverage: 1.81% of modified lines in pull request are covered.

Additional details and impacted files
@@              Coverage Diff              @@
##             master    #2238       +/-   ##
=============================================
- Coverage     73.03%   14.98%   -58.06%
+ Complexity    10381     2347     -8034
=============================================
  Files           749      749
  Lines         56830    56905       +75
  Branches       6984     7003       +19
=============================================
- Hits          41508     8525    -32983
- Misses        13081    47882    +34801
+ Partials       2241      498     -1743

Impacted Files
Coverage Δ

...n/java/com/github/ambry/commons/ServerMetrics.java
0.00% <0.00%> (-98.31%)
:arrow_down:

...ambry/messageformat/MessageSievingInputStream.java
0.00% <0.00%> (-96.91%)
:arrow_down:

.../java/com/github/ambry/protocol/AmbryRequests.java
0.00% <0.00%> (-80.89%)
:arrow_down:

...main/java/com/github/ambry/server/AmbryServer.java
0.00% <0.00%> (-81.74%)
:arrow_down:

...a/com/github/ambry/server/AmbryServerRequests.java
0.00% <0.00%> (-89.90%)
:arrow_down:

...va/com/github/ambry/clustermap/MockDataNodeId.java
39.75% <100.00%> (-45.43%)
:arrow_down:

.../src/main/java/com/github/ambry/router/Router.java
0.00% <0.00%> (-100.00%)
:arrow_down:

.../src/main/java/com/github/ambry/store/Message.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...rc/main/java/com/github/ambry/named/PutResult.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...rc/main/java/com/github/ambry/quota/QuotaMode.java
0.00% <0.00%> (-100.00%)
:arrow_down:

... and 594 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.


Access to on premisse LLM

Expected Behavior
I would like to bind LoLLMs to a LLM that is deployed on premisse in our organization.
Current Behavior
It seems there is no binding allowing to set an endpoint, an engine, a version, a token ...
Apologies if I missed anything in the doc.

Hi. Ofcourse this is possible.
I have done this multiple times in multiple places.
You have many options.
For example, you can install a vllm server on a server then on lollms, select vllm binding, and after installing it, you can setup the link to your server as well as the parameyters like the key if you have one etc. Then you'll be able to access your served models on your client pc on lollms.
You can do the same using ollama as server.
Best regards


Add null checks to UiPin methods

This changelist adds null checks to the UiPin::addConnection and UiPin::deleteConnection methods in the Graph Editor, providing additional protection against crashes.

Looks great to me!


VR Support

I'll try and take a look myself but I'm trying to find a raymarcher that blits directly to the camera that does work in VR to use as an example. Most tend to use a quad parented to the camera.

The alternative being a quad in any position where the vertex shader repositions it at the camera?

I think the alternative being the solution used here, which is elegant as hell - blit within OnRenderImage, passing the camera matrix, works perfectly with the scene camera. Thank you for making and tutorial-izing this @Flafla2 <3
Any idea what's missing to make this work in VR? It breaks Single Pass, which may be a complicated fix, but Multi Pass should be easy in theory. In Multi Pass the matrix is slightly off, but symmetrically for each eye, the camera does pass a separate matrix for each.
I wasn't able to find a correct matrix in the APIs. I tested it on a Vive, with Camera.stereoActiveEye, and Camera.GetStereoViewMatrix(StereoscopicEye eye). I wonder if the SteamVR_Camera is doing something different on render. But I'm a bit out of my depth with these SteamVR / open XR APIs. For instance I was able to get a projection matrix from SteamVR (with HMDMatrix4x4ToMatrix4x4(SteamVR.instance.hmd.GetProjectionMatrix(evrEye ...)), but not a view matrix. Any ideas?


October 05 meet-up session

Summary

Deadline
Date/time is Thursday 05 October 12-13:30
What needs to be done?

[x] Create and confirm agenda and google notes doc
[x] Include details in newsletter
[ ] Send out reminder email

Who can help?

Please put any suggestions for the agenda in the comments below this issue

Updates
Google doc for notes including agenda is here: https://docs.google.com/document/d/11krYiT2zHcKKkfAUMYIvMkq4FitmEQhGUmOlQNlOMHI/edit

this is done and went well! 🚀


[FLINK-13708] [table-planner-blink] transformations should be cleared after execution in blink planner

What is the purpose of the change
transformations should be cleared after execution in blink planner
Brief change log

clear transformations in ExecutorBase

Verifying this change
This change added tests and can be verified as follows:

Added testExecuteTwiceUsingSameTableEnv to verify this bug

Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): (yes / no)
The public API, i.e., is any changed class annotated with @Public(Evolving): (yes / no)
The serializers: (yes / no / don't know)
The runtime per-record code paths (performance sensitive): (yes / no / don't know)
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (yes / no / don't know)
The S3 file system connector: (yes / no / don't know)

Documentation

Does this pull request introduce a new feature? (yes / no)
If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)

I have a bit different question. Why ExecutorBase buffers the transformations in the first place? It violates the contract of Executor#apply. The assumption is that apply method will really apply the transformations to the ExecutionEnvironment that was provided. We do the buffering in the TableEnvironmentImpl already.
The way it is implemented right now. Such program, which is a valid program will produce no results:
StreamExecutionEnvironment senv = ...
StreamTableEnvironment tEnv = StreamTableEnvirornment.create(senv)
tEnv.sql(...)
senv.execute();

I have a bit different question. Why ExecutorBase buffers the transformations in the first place? It violates the contract of Executor#apply. The assumption is that apply method will really apply the transformations to the ExecutionEnvironment that was provided. We do the buffering in the TableEnvironmentImpl already.
The way it is implemented right now. Such program, which is a valid program will produce no results:
StreamExecutionEnvironment senv = ...
StreamTableEnvironment tEnv = StreamTableEnvirornment.create(senv)
tEnv.sql(...)
senv.execute();

different from the implementation of StreamExecutor in flink planner, Executors in blink planner will generate StreamGraph first and execute the stream graph directly. StreamExecutor in blink planner should buffer the transformations to generate a whole StreamGraph (StreamGraph does not support merge now) if the operation translation is eager (using StreamTableEnvirornment). To unify the implementation for batch and stream executor, we defined a buffered transformations in ExecutorBase.
one approach to avoid buffering transformations is StreamExecutor in blink planer does not generate StreamGraph any more, and keep the same implementation as StreamExecutorin flink planner. andBatchExecutorgenerateStreamGraphinapply` method

@godfreyhe I see the problem. TBH I would prefer the approach you suggested more than how it currently is implemented. It supports the old behavior.

@godfreyhe I see the problem. TBH I would prefer the approach you suggested more than how it currently is implemented. It supports the old behavior.

I find another problem today: if explain twice, the second result contains the first result, or if execute after explain, the StreamGraph to execute also contains transformations generated in explain. This bug exists in both planners.
the latest fix is: generate StreamGraph not through StreamExecutionEnvironment but StreamGraphGenerator directly. This approach could avoid StreamExecutionEnvironment buffer the transformations added in explain method or in execute method, and could generate clean StreamGraph. This will be fixed in blink planner now.

Could we take a step back here and think once again when/what/why is buffered?
The current state of this PR does not fix the StreamTableEnvironment with blink planner. You can not call execute on StreamExecutionEnvironment. Also I don't understand why should we throw exception in explain method.  I will list from my understanding:

operations should be buffered in unified table env. In order to reuse the shared part in the graph. And should be cleared after execute is invoked. (This is what we do currently.)
transformations should be buffered in Executor not in StreamExecutionEnvironment in blink planner. Blink batch need to set additional properties on the transformation before execute. We should also clear the buffered transformations after execute is invoked.

The current state of this PR does not fix the StreamTableEnvironment with blink planner. You can not call execute on StreamExecutionEnvironment

For StreamTableEnvironment, I don't think we can fix this issue. Because StreamExecutionEnvironment doesn't support execute multiple times. Is that right?

Also I don't understand why should we throw exception in explain method.

Currently, StreamTableEnvironment.explain returns nothing even if we applied some queries before. I think throwing an exception will be better to warning users this method is not work properly currently. We can return the buffered logical/physical plan in the future, but it needs more work.

I get the problems with BatchExecutor I can't think of a better solution for now.
Regarding the:

For StreamTableEnvironment, I don't think we can fix this issue. Because StreamExecutionEnvironment doesn't support execute multiple times. Is that right?
Could you elaborate a bit more. Why do we need that?

Hi @dawidwys ,
After revisiting the code of StreamExecutionEnvironment, I find that I'm wrong. StreamExecutionEnvironment supports to execute multiple times, because it will clear the buffered transformations.
Regarding to why we need execute multiple times:

we didn't forbid to execute multiple times in the JavaDoc of execute method.
StreamExecutionEnvironment also supports execute multiple times.
We are doing the benchmark for TPC-DS queries and the total time is one of the important metrics. Meanwhile, we will collect and register some statistics in the built-in catalog, this will takes some time because it will spawn some jobs to calculate ndv, row-count, etc... If the TableEnvironment doesn't support execute multiple times, we have to re-calculate the statistics and re-construct TableEnvironment, this will prolong the total benchmark time.

There are three issues:

explain() + execute()    ==> will execute the job twice
explain() + explain()     ==> will display two same physical plans
execute() + execute()  ==> will execute the job twice in the second execute

Blink planner has all of three issue. And flink planner has 1) and 2) issues.

I get that we should change sth. I am trying to understand if we need the whole buffering business in executors. I understand now that we need it in BatchExecutor. But I really see no reason why Blink StreamExecutor should behave differently from Flink's.
Ad. 1 & 2 The problem is not in how the Executor is implemented (at least in the Flink planner), but because of the TableSource interface. Which always adds transformation to a StreamExecutionEnvironment. Generally speaking explain should be stateless.
Just to sum it up a bit. I think changes in this PR are fine for now to improve the BatchExecutor. However, unless you have a good reason for it, and I haven't heard one so far, I would really prefer that the StreamExecutor does not buffer the transformations, because in the end I think we should aim for all the Executors to be stateless.

However, in order to make explain stateless, we have to get a StreamGraphGenerator to generate StreamGraph ourselves. We can't get StreamGraphGenerator from the underlying env. That's why we construct the StreamGraphGenerator in Executor to do this. And we would like to make sure the generation behavior is the same between explain  and execute, so we buffers the transformations to generate StreamGraph to submit.

No you don't buffer in this PR. In the explain method you just generate StreamGraph from the transformations you got in the explain method. Which is correct.
There is actually one more thing that I don't necessarily like about it. The Planner has to construct the Executor which violates the clear seperation of logical/relational planning and physcial/execution planning. How about we add a method explain to the Executor as well. This method would explain the physical part.

No you don't buffer in this PR in the explain method. In the explain method you just generate StreamGraph from the transformations you got in the explain method. Which is correct.
Yes. We didn't buffer transformations for explain method.

There is actually one more thing that I don't necessarily like about it. The Planner has to construct the Executor which violates the clear seperation of logical/relational planning and physcial/execution planning. How about we add a method explain to the Executor as well. This method would explain the physical part.
How about  add a String explainExecutionPlan(List<Transformation<?>>) ?
explain is too generic and it doesn't return logical parts.

so sorry for late reply, and thanks all guys for the suggestion. The implementation of Blink StreamExecutor is same with Flink StreamExecutor now, except two added getStreamGraph methods: one is for explain method, and another is for sql client.

Any update on this PR. This is an very critical issue which will cause incorrect job result (multiple copies of result if the same sql run multiple times via the same env)

when/what/why is buffered? I will list from my understanding:

operations should be buffered in unified table env. In order to reuse the shared part in the graph. And should be cleared after execute is invoked. (This is what we do currently.)
transformations should be buffered in Executor not in StreamExecutionEnvironment in blink planner. Blink batch need to set additional properties on the transformation before execute. We should also clear the buffered transformations after execute is invoked.

Hi @wuchong @godfreyhe , I agree with @dawidwys , we should let Executor stateless (Except its field StreamExecutionEnvironment).
Why we need buffer transformations? Just add them to StreamExecutionEnvironment?
You can take a look to StreamExecutionEnvironment.execute(StreamGraph), this will clear transformations in StreamExecutionEnvironment. I think this can solve your concerns?
So I think we don't have reason to buffer transformations?

@JingsongLi @dawidwys thanks for the suggestions. Yes, Executor does not need buffer transformations, can be made stateless, simple and clean. We could add a utility class to generate StreamGraph for explain, and put setBatchProperties method (will be used for both executor and explain) in this utility class. I will update the pr soon.

@zjffdu sorry for late update

No problem, thanks for the fix. @godfreyhe

@godfreyhe It looks like there's no other concerns, could we get it merged soon after resolving the conflict ?

cc @dawidwys @KurtYoung  @wuchong @JingsongLi , have any other suggestions ?

updated based on comments

LGTM. +1

thanks for all guys' help!

Running travis on my own repo. Will merge this when it is passed.


Add Log In button enhancement

The is sign in with github button but no login button . So can add the feature in the project.

please assign me this work!!
@prakhartiwari0

@prakhartiwari0 Thankyou!!

This project doesn't use any backend so I don't think this issue makes any sense.


MinkowskiEngine install

python setup.py install --blas_include_dirs=${CONDA_PREFIX}/include --blas=openblas should add  --force_cuda

Thank you for your testing and feedback.
For installation guidance on MinkowskiEngine, please  refer to the official repository for more detailed information.


Production build fails with "error: "version" is a required argument."

Description
building for production fails
Expected behavior
building for production should finish
Actual behavior
After runing production build I get
18:30:29 - error: "version" is a required argument.
  at Object.getArg (/Users/uros/Sites/wandera/node_modules/source-map/lib/util.js:24:11)
  at SourceMapConsumer.BasicSourceMapConsumer (/Users/uros/Sites/wandera/node_modules/source-map/lib/source-map-consumer.js:288:22)
  at new SourceMapConsumer (/Users/uros/Sites/wandera/node_modules/source-map/lib/source-map-consumer.js:22:7)
  at prepareSourceMap (/Users/uros/Sites/wandera/node_modules/brunch/lib/fs_utils/generate.js:130:48)
  at processJob.then.optimized (/Users/uros/Sites/wandera/node_modules/brunch/lib/fs_utils/generate.js:151:17)
  at process._tickCallback (internal/process/next_tick.js:103:7)

Environment

Brunch: 2.10.8
Node.js: v6.7.0
NPM: 3.10.3
Operating system: osx

package.json contents
"devDependencies": {
    "autoprefixer": "^6.7.3",
    "brunch": "^2",
    "coffee-script-brunch": "^2",
    "coffeelint-brunch": "^2",
    "digest-brunch": "^1.4.0",
    "git-revision": "*",
    "groundskeeper-brunch": "0.0.4",
    "handlebars-brunch": "^2",
    "karma": "~0.13",
    "karma-chrome-launcher": "~0.1",
    "karma-coffee-preprocessor": "~0.2",
    "karma-coverage": "~1.1",
    "karma-firefox-launcher": "~0.1",
    "karma-junit-reporter": "~1.1",
    "karma-mocha": "~1.2",
    "karma-mocha-reporter": "~2.2",
    "karma-phantomjs-launcher": "~1.0",
    "karma-safari-launcher": "~0.1",
    "keyword-brunch": "^2",
    "mocha": "~3.1",
    "phantomjs-prebuilt": "~2.1",
    "postcss-brunch": "0.5",
    "sass-brunch": "^2",
    "uglify-js-brunch": "^2"
  },

Other useful files, when present (log, bower.json etc.)
Latest version I used was 2.8.x and no errors with the same codebase and configuration

Hi, @urosgruber!
I'm trying to help you with your problem. Could you help me and create a repo that reproduce your troubles with Brunch?

@denysdovhan could be a bit of a problem as code is closed source. I'll try and see if it's code related issue.

Make sure you're using latest postcss-brunch.

Ok. Found it. it's groundskeeper-brunch.
Here is my package.json
{
  "author": "Foo",
  "name": "foo",
  "description": "",
  "version": "1.0.0",
  "private": true,
  "repository": {
    "type": "git",
    "url": ""
  },
  "engines": {
    "node": "0.8.x",
    "npm": "1.1.x"
  },
  "scripts": {
    "build": "brunch b",
    "build:prod": "brunch b -p"
  },
  "dependencies": {
    "backbone": "1.1.2"
  },
  "devDependencies": {
    "brunch": "^2",
    "groundskeeper-brunch": "0.0.4"
  },
  "license": "BSD"
}

and config.coffee
exports.config =
    paths:
        public: 'public'
        watched: ['app/javascripts']

    fileListInterval: 300
    sourceMaps: true

    ## Files
    files:
        javascripts:
            joinTo: 'app.js'

That is all you need. If groundskeeper-brunch is removed production build is fine as well. FYI non production works always.

Try https://babeljs.io/docs/plugins/transform-remove-console. groundskeeper is so 2013.

@urosgruber babel-brunch with transform-remove-console plugin would be much more better solution:
module.exports = {
  plugins: {
    babel: {
      plugins: ['transform-remove-console']
    }
  }
}


Student Confirms Profile information

As a (Student) I want (confirm my profile information) So That (so that I can verify that my address is current on my application)
Additional Context

The Profile information is the source of Truth on the Student Profile- not the application

Acceptance Criteria

[ ] Given (A Student), When (A student initiates an application), Then they are confirming that Read-Only view of both BC Services Card and SABC Profile Information is correct.
[ ] Given (A student), When (cannot confirm the Profile Information), Then (they will be redirected to the Student Profile Section).
[ ] Given (A student), When (they confirm their profile info is correct), Then (they select 'Yes),
[ ] Given (A student), When (they confirm their profile info is incorrect), Then (they select 'No, I need to update my profile information')
[ ] Given (A student), When (they are viewing the profile information ), Then (it is NOT displayed as editable (its greyed out) for both Services Card and SABC Profile Information

Technical Tasks

[ ] Display the entire student profile as a read-only. Probably creating a component to hold this.
[ ] Starting saving to DB the fields that are not being saved right now (e.g. DOB and Gender)
[ ] Update the new fields on DB during user synchronization after the login.
[ ] Expose in the API an endpoint to retrieve all the student data needed to populate all the fields.


add numb as number prefix for prose formatting

make it consistent with the formatter_immune capture in formatters.py, which has both numb and numeral

We deliberately didn't do this when adding this feature to avoid misrecognitions: https://github.com/knausj85/knausj_talon/pull/737#issuecomment-1013947323
eg. "he was numb to the pain" --> "he was 2 the pain"
I'm not super firm on this, if you want to argue that it's worth it or that the misrecognitions won't be super common, go ahead.

I see. Then I don't see why the formatters allow numb - of course it is less likely to have "numb" in a variable name I guess, but it still feels unclean and possibly confusing to me.
Like that "scratch that" and "nope that" both work for prose/text.talon, but only the first works in dictation mode.
But I can see if these little inconsitences are not such a big of a deal, but I feel like they are making it more difficult than it has to be.

I'd be inclined to make this change (so numb two/to/too -> 2), I reckon the sub-phrase is uncommon enough that the benefits of consistency outweigh the potential costs.
I just tried this branch with Conformer D (beta), and Conformer B (public) and both of them actually continue to output 'He has become numb to it' (and some other variants) for some reason. For other numbers it outputs the digits as you'd want.
So empirically it sounds like this isn't an issue as well (at least with my accent).

Ok, I guess if somebody has issues on a regular bases with that they can still remove it, but adding it is imho always more difficult because you first have to know about it.

I'm going to leave this open for another week to see if anybody else wants to weigh in, and then plan to merge it.


Cannot use fail_point! 3 arguments macro without importing it

Describe the bug
Cannot use full name qualification for fail_point! macro in the 3 arguments case
To Reproduce
Just try to compile:
fail::fail_point!("fail-point-3", enable, |_| {});

And you'll get:
error: cannot find macro `fail_point` in this scope
   --> my_code.rs:10
    |
10 |                     fail::fail_point!("fail-point-3", enable, |_| {});
    |                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |
    = note: consider importing this macro:
            fail::fail_point
    = note: this error originates in the macro `fail::fail_point` (in Nightly builds, run with -Z macro-backtrace for more info)

Expected behavior
You should be able to use the macro without importing it with use
Additional context
Looks like the issue is here: https://github.com/tikv/fail-rs/blob/6645f1767b4d0adbe454a07c25e30224b1b08dbd/src/lib.rs#L841
The recursive macro invocation should look like this:
$crate::fail_point!($name, $e);

Good catch! Would you like to send a PR to fix it?

Hi @BusyJay, here is the PR: https://github.com/tikv/fail-rs/pull/66

Hi @BusyJay, is there any plan to release a new version of this crate? There's a bunch of fixes since the last release :)

Thank you @BusyJay for the release!


ci: add suggestions for typos

Description of changes:

Call-outs:

Testing:

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

GHA annotations don't support markdown so this won't work, unfortunately