refactor: Update Fether menu for taskbar mode on non-macOS (add 'Quit', remove 'Close Window', remove 'Minimize')

Refactor Fether menu code

Add File > Quit to Fether menu when not using macOS

Remove the Fether menu items "Window  > Close Window" and "Window > Minimize" when running in taskbar mode on macOS, since we are only allowing user to either blur the window, hide and re-open it by clicking the taskbar icon, or quit

Remove the Fether menu tab "Window" entirely when running in taskbar mode and not using macOS (since it would on'y have "Close Window" and "Minimize" menu items, and we don't need either of them, see previous dot point for justification

Note that subsequent to this PR we need to make the Fether context menu appear with a "frameless" Fether window on all OS

Add File > Quit to Fether menu when not using macOS

I don't quite understand. Since it's frameless, there is no menu on Linux (as it doesn't work, and that's fine because there's no need for it).
But then there is no "File>Quit" as well. Hence https://github.com/paritytech/fether/issues/438

Add File > Quit to Fether menu when not using macOS

I don't quite understand. Since it's frameless, there is no menu on Linux (as it doesn't work, and that's fine because there's no need for it).
But then there is no "File>Quit" as well. Hence #438

It's possible to have the context menu appear as a popup when you right-click on a "frameless" Fether window, and even have the context menu appear as a popup when you right-click on the system tray icon, as shown in this screenshot

I think @axelchalon's comment https://github.com/paritytech/fether/pull/433#issuecomment-465592005 was referring to the fact that there wasn't a "Quit" menu item at all for Windows and Linux in the Fether menu, we only had one on macOS.
I've changed this PR back to being in-progress.
Previously when I used tray.setContextMenu in this commit to show the context menu popup on right-click, I was able to get the context menu popup to appear, but when I clicked the menu items it didn't respond https://github.com/paritytech/fether/pull/332/commits/8ce7addb3b4c8b7b5f2eeca5460b381ef1bc6168
Whereas today I was able to show the context menu popup on right-click, and the items worked when using remote approach in the front-end instead, as mentioned here https://electronjs.org/docs/api/menu#render-process
I think we need the Fether context menu on Windows and Linux, and I'm going to try and get the right-click popup context menu to work.

@Tbaut I've pushed commits so that on Linux and Windows if you click the system tray icon it shows a context menu. It all works on both Ubuntu Gnome OS and Linux Mint OS. There's just an issue on Windows OS where clicking the menu items in the "Edit" and "Window" sections of the menu doesn't work.
Linux Mint

Ubuntu Gnome

Windows 10

Update on Windows issue where context menu 'Edit' and 'View' menu tab menu items not working:
If I configure the menu using webContents I've been able to get all the menu items in the 'View' tab working on Windows 10. I've also used the same approach to get all the menu items in the 'Edit' tab to work on Windows 10, which also requires me to manually import the instance of Fether app (i.e. fetherAppInstance and then run fetherAppInstance.win.focus() when tray.on('click' is called, which is one of the event listeners that fortunately works on Windows!

Thanks a lot for the thorough testing.
TBH I really think it's overkill to have so many try menus. All I need as a user is "Quit Fether" and it's now "hidden" behind "Electron" which as a random user have no idea what this means :nerd_face:. Also I do not expect to have edit, copy, paste etc.. in such a menu. I've never seen this on other applications.
What Riot does and I believe we should do as well (keep it simple):

Tray icon:

Show/Hide Riot
Quit

Left click on a selection (I don't think we really need that for now, shortcut are enough):

Cut
Copy
Paste
Paste without style (no need for us)
Select all (no need for us)

Let's discuss all this beforehand in an issue so that we only implement what makes a consensus.

Thanks a lot for the thorough testing.
TBH I really think it's overkill to have so many tray menus.

I think we need all the menu items that we already have, but without "Force Reload", "Actual Size", "Zoom In", "Zoom Out", and "Toggle Full Screen" menu items in the "View" menu tab. I think we additionally need "Undo" and "Redo" menu items in the "Edit" menu tab.
In the latest commits that I've pushed I've only included the menu items that I think are needed, based on what I think users may need, and removed ones that I don't think they'll need. Please try it out

All I need as a user is "Quit Fether" and it's now "hidden" behind "Electron" which as a random user have no idea what this means 🤓.

We've always that menu tab on macOS prior to implementing the taskbar app. In development it appears with the menu tab name "Electron" and in production it appears with menu tab name "Fether", because it's using the app's name. I've renamed it to just "File" to avoid further confusion (which is consistent with what it's called in most applications)

Also I do not expect to have edit, copy, paste etc.. in such a menu. I've never seen this on other applications.

I think we need all the "Edit" menu items. For example if I'm using Fether in a VM, then pressing shortcuts into input fields may not work, so I'd be grateful as a user if the app developer had added an Edit menu with the menu items that we've included. I don't think we should just copy how other applications do it, they may not have included features like this that are helpful to some users, and they don't get in the way of users that don't user the features either.
I also think we only need "Reload" and "Toggle Developer Tools" menu items in the "View" menu tab

@Tbaut Note that I've pushed this commit https://github.com/paritytech/fether/pull/440/commits/f8b72f85435df9d9de8638089847824005431b74 that removes the window resize that's not necessary anymore since we're using "frameless" windows on all OS. This should hopefully address the issue you found here https://github.com/paritytech/fether/pull/436#issuecomment-465991913

@Tbaut Thanks for your great suggestions, on Linux and Windows I agree it would be good to remove the "File > Exit" and "Help > Learn More" and replace them with "About" and "Quit" menu items right at the bottom of the context menu. Additionally displaying the "Edit" menu items when the user right-clicks the Fether window (i.e. easy access to Cut/Copy/Paste).
I'll proceed with making those changes

Update: I've added About and Exit to the bottom of the context menu on Windows and Linux, and removed File and Help menus

@Tbaut I've pushed a commit that displays the context menu when you "right-click" in the Fether window that contains the Edit, View, and Help submenus, and Quit. I know we discussed only having the Edit menu items appear when you "right-click" in the Fether window, but I think it's better also having the the View and Help submenus and Quit item there too (rather than adding confusion by making the user have to click the "tray" icon instead to access those other menu items). So on that basis I've also removed the feature that also displayed the same context menu when you clicked the "tray" icon, so now clicking the tray icon just shows/hides the Fether window as it did before.
Please let me know what you think?

Due to not being able to avoid this and the fact that right-click on the tray icon doesn't work on Linux either, I've proposed that for all OSes

I've since discovered that we can show the tray menu with "Show/Hide Fether" and "Quit" items by running tray.popUpContextMenu(); after we set the context menu that we want to use i.e. tray.setContextMenu(fetherApp.contextTrayMenu.getMenu()); in the tray.on('right-click' listener

Linux:

Left and right click on the tray icon shows:

Show/Hide Fether
Quit

This works!

Mac and Windows:

Left click on the tray icon directly toggles Fether

Right click on the tray show:

Show/Hide Fether
Quit

On macOS, when we configure the tray icon to display the context menu, it only runs the code in the 'right-click' listener when you either left-click or right-click the tray icon (i.e. 'click' and 'double-click' do not work). I think it's related to this issue.
But users can just select "Show/Hide Fether" and it'll show/hide the Fether window
So it ends up being the same functionality as on macOS.
On Windows, it all works the way we want it to (i.e. the left-click the tray icon toggles Fether directory and right-click shows the context menu with "Show/Hide Fether" and "Quit" items)


feat: request action validation framework created, and implemented on sign and submit endpoint

closes #482
closes #515
new context, and validation dependency concept, currently only applying to the sign endpoint, can extend to additional endpoints.
also removed some of the tasks code since we no longer going with that concept; allowing for some daos to not be detached.

First thank you for cleaning up the task_actions! Left it in there just in case we revisited it but think that ship has sailed.
If someone fat fingers a validation function or adds one to the list without adding the corresponding impl what happens? Might be good to add a check and error gracefully?

at the moment, a non-existent check gets ignored, so it still runs through the remaining existing checks; although I should definitely at least log it.

First thank you for cleaning up the task_actions! Left it in there just in case we revisited it but think that ship has sailed.
If someone fat fingers a validation function or adds one to the list without adding the corresponding impl what happens? Might be good to add a check and error gracefully?

at the moment, a non-existent check gets ignored, so it still runs through the remaining existing checks; although I should definitely at least log it.

Ok yeah I see where you're filtering now.  So yeah I think that would be the only thing I would add here.  Seems like if there is supposed to be a check we want to do, as configured by the environment, but we don't actually have that check we'd want to flag it somewhere.  Not sure if that should fail validations?  Doubt it's a case we'll come across a lot.

Looks good. Hopefully will come into play for more endpoints in the future

Yeap, I think some endpoints can leverage this already, will get a ticket created and check them out.


Introduce getSynthesizedDeepClone

During refactoring, it seems quite common to require that a node appear in more than one place in the rewritten tree.

Apparently, this is a public API change.  What's the correct way to handle that?

If it's desirable, just accept the new API baselines.

Or mark it @internal like getSynthesizedClone.

@andy-ms's right - it was an oversight on my part.

@rbuckton I believe I've addressed your feedback.

@rbuckton - I believe most/all callers will want to copy trivia. Does that mean deep-clone should actually be built on getMutableClone, rather than getSynthesizedClone?

I'd rather not use getMutableClone as I'd like to eventually get rid of that function. You could always call setTextRange in your getDeepSynthesizedClone to copy positions.

@rbuckton Could I just refrain from clearing pos and end?  (I assume I still have to clear parent.)

As of this PR, the positions don't make a difference.  I'll add a test once https://github.com/Microsoft/TypeScript/pull/18931 is merged.

@rbuckton signed off offline.


Can't deploy bookinfo demo: ran out of forwarding rules limit

I've completed deploying Istio with instructions here : https://istio.io/docs/tasks/installing-istio.html
Now I'm trying to deploy the BookInfo app: https://istio.io/docs/samples/bookinfo.html
I executed the following on an empty GKE cluster (just grafana stuff and istio installed):
$ kubectl apply -f <(istioctl kube-inject -f bookinfo.yaml)
service "details" created
deployment "details-v1" created
service "ratings" created
deployment "ratings-v1" created
service "reviews" created
deployment "reviews-v1" created
deployment "reviews-v2" created
deployment "reviews-v3" created
service "productpage" created
deployment "productpage-v1" created
ingress "gateway" created

Now I have the following services:
NAME            CLUSTER-IP      EXTERNAL-IP      PORT(S)                       AGE
details         <IP_ADDRESS>    <none>           9080/TCP                      13m
grafana         <IP_ADDRESS>    <IP_ADDRESS>   3000:32365/TCP                15m
istio-egress    <IP_ADDRESS>     <none>           80/TCP                        15m
istio-ingress   <IP_ADDRESS>    <IP_ADDRESS>   80:31456/TCP                  15m
istio-manager   <IP_ADDRESS>   <none>           8080/TCP,8081/TCP             15m
istio-mixer     <IP_ADDRESS>   <none>           9091/TCP,9094/TCP,42422/TCP   15m
kubernetes      <IP_ADDRESS>     <none>           443/TCP                       30m
productpage     <IP_ADDRESS>   <none>           9080/TCP                      13m
prometheus      <IP_ADDRESS>   <pending>        9090:31569/TCP                14m
ratings         <IP_ADDRESS>    <none>           9080/TCP                      13m
reviews         <IP_ADDRESS>   <none>           9080/TCP                      13m
servicegraph    <IP_ADDRESS>   <pending>        8088:31123/TCP                14m

as you can see some of them are <pending> on external IP. When I did describe on the service I saw
Events:
  FirstSeen	LastSeen	Count	From			SubObjectPath	Type		Reason				Message
  ---------	--------	-----	----			-------------	--------	------				-------
  12m		1m		4	service-controller			Normal		UpdatedLoadBalancer		Updated load balancer with new hosts
  13m		45s		8	service-controller			Normal		CreatingLoadBalancer		Creating load balancer
  13m		24s		8	service-controller			Warning		CreatingLoadBalancerFailed	Error creating load balancer (will retry): Failed to create load balancer for service default/servicegraph: failed to create forwarding rule a1a665b1534f911e7ae1242010a800fd: googleapi: Error 403: Quota 'FORWARDING_RULES' exceeded. Limit: 15.0, quotaExceeded

However my ingress has an IP address:
$ kd ing
Name:			gateway
Namespace:		default
Address:		<IP_ADDRESS>
Default backend:	default-http-backend:80 (<IP_ADDRESS>:8080)
Rules:
  Host	Path	Backends
  ----	----	--------
  *
    	/productpage 	productpage:9080 (<none>)
    	/login 		productpage:9080 (<none>)
    	/logout 	productpage:9080 (<none>)
Annotations:
Events:	<none>

Note that the ingress doesn't have any GCLB url forwarding rules and when I go to https://pantheon.corp.google.com/networking/loadbalancing/loadBalancers/list I don't see the LB created for the ingress. The external IP exists but it doesn't do anything.
I realized I had to clean up some forwarding rules (screenshot below) from my previous clusters to open up some space. (Now I'm waiting for things to get fixed...)
I am aware that this is a GCP problem and not istio, but is there anything the docs can do to remediate this?

We can use a kube proxy tunnel to ingress service to avoid allocating an external IP.
Anycase, this deserves a mention in the installation instructions.
@andraxylia

GKE allocates an external IP Address, but it fails to add the the Google Load Balancer. This is outside of Istio. It may be because you exhausted your quota.

I'll close this as this was mostly my old clusters leaking LBs/Forwarding Rules/IPs.


Explose two callbacks in TGVE: onViewStateChange and onStateChange

This is the first of API callbacks. Both are important for better integration of TGVE to applications and are overdue.

The tests for these changes have been focused on an "OSM buildings viewer" using osmbuildings api to pull building boundaries as GeoJSON and vis them in TGVE. The throttling applied to the "update URL" might need to change but will have to wait until issues are raised.

onStateChange is almost a reverse of what current select API does and can be seen in this PR.

[ ] tests should avoid React 18 for now
[ ] make sure when custom data is loaded it takes priority over props. So priority should be 1. URL params 2) loaded data via "add data" 3) props etc?


Add feature to dynamically set per-row CSS class or classes

Is your feature request related to a problem? Please describe.
There are use cases where we need to style individual rows of the table based on some logic.
Describe the solution you'd like
Developers should be able to provide a callback function to the table component that gets a row's data as input and returns a class or class list (separated by spaces) as a string literal.

Fixed in #52


How can I get "./base_50.pt"?

In the classfication/main.py,there is a parameter named ckpt_dir_fg in line26，the default value of it was ./base_50.pt. How
can I get this file?

So can you please share how to get the "./base_50.pt"? I face the same problem too.

You are correct @zhangziyi1670. Thanks for your contribution!


ml.g4dn.xlarge limit issue

Hi, thank you for your awesome example! Im trying to run it and I get the following error:
ResourceLimitExceeded: An error occurred (ResourceLimitExceeded) when calling the CreateEndpoint operation: The account-level service limit 'ml.g4dn.xlarge for endpoint usage' is 0 Instances, with current utilization of 0 Instances and a request delta of 1 Instances. Please contact AWS support to request an increase for this limit.
I have seen this thread about the same problem https://github.com/awslabs/amazon-sagemaker-examples/issues/307#issuecomment-545707229
and when I try to request a SageMaker service limit upgrade for the "SageMaker Notebooks" option, ml.g4dn.xlarge is not on the list. I was wondering how you managed to get it to run with ml.g4dn.xlarge? Is there any alternative choice?

@aidamanzano, in this notebook you're deployingYolo model on Sagemaker endpoint. So, you need to request limit increase under "Sagemaker Hosting" category.

Thank you for your response, I've searched under the hosting category and I can't find ml.g4dn.xlarge as one of the options. Which one should I pick so that I don't run into the Resource Limit error?


parameter_input exception

if the step is type parameter_input, than it has no tool id etc ...

just need to change the type/format of parameter_input


Unify the representation of dynamic variables in logs and exceptions to use a square bracket to wrap them for log analysis

: isn't suitable for a message with multiple variables. For example, Failed to set the field "?" to the value: ?, and their representation isn't consistent. It's better to use Failed to set the field [?] to the value [?] consistently.


decoding of HEVC operation point descriptor.

this version treats the HEVC Profile Tier Level information (the sets of 96 bits) opaquely - a future update will dissect these

Why did you remove the AVS3 descriptor from the test?
The idea is to never remove anything. Otherwise your AVS3 code will no longer be tested in the future and could become broken without being noticed. Something that is proved correct once remains forever to make sure it is never broken.

I commented out the other descriptors while I was trying to debug another issue - they are reinstated now along with the new HEVC_operation_point descriptor.


Add multiplication

resolves #3

Oh I see some build fails, there is a bad test in the suite can you fix that?


No communication two ESP32 with SN65HDV230 module

Hello,
I have two esp32 (DOIT devkit esp32) and two module with SN65HDV230. I used yours library. On one is load program CANsender and on second esp32 is program CANReceiver. First ESP32 sending me on serial message "sending packet...done" . My configuration is same like yours. I dont see on second ESP32 no messages. I testing change speed and pins and no reaction.
I have more modules(ESP32 and SN65HDV230) and all of it is the same.
Please help me.
Thank you

Old thread but FYI
MCP2551/2561 has 5V CMOS level interface with MCU and it violate VIH(min).
If you use 3.3VIO MCU, just use MCP2562 which has separate I/O supply pin.


'buildah from' error on image argument placing is unclear

Description
Doing  buildah from scratch --name builder yields:
too many arguments specified
ERRO[0000] exit status 1

The error being raised is fine, but the message should instead be something like "image name is not the last argument provided".
Steps to reproduce the issue:

See description

Describe the results you received:
See description.
Describe the results you expected:
An error pointing out that the image needs to be the last argument.
Output of buildah version:
# buildah version
Version:         1.4-dev
Go Version:      go1.9.7
Image Spec:      1.0.0
Runtime Spec:    1.0.0
CNI Spec:        0.3.1
libcni Version:  v0.6.0
Git Commit:      00dd72aa
Built:           Thu Aug 16 13:35:08 2018
OS/Arch:         linux/amd64

Reported by "fatpointer" on IRC.

we can close this.


Manually re-define functions under function module with proper type definitions

Functions under the function/_generated.ts module are produced programmatically without type definitions. So we need contributors helps to re-define these manually with proper type definitions like function/buffer.ts.
How to (partially) contribute to this issue

Add (or update) a module in function and write proper function definitions

Write proper type definitions
Write proper document comments
Write unit tests for the functions

Add the module to _manual.ts if missing
Execute make gen to re-generate _generated.ts so that manually defined functions are removed from _generated.ts
Create a PR for that and link to this issue

Please use definitions of the following while those are the minimum versions which denops.vim support

Vim 8.1.2424 eval.txt
Neovim 0.4.0 eval.txt

[ ] cursor-functions @kuuote


Alias is not used by the _find for arguments parsing

When looking for an argument, the _find function doesn't search for an alias registered
import vyper
import argparse
import os

v = vyper.Vyper()

v.automatic_env()
v.register_alias('arg', 'name')
v.bind_env('name', 'ENV_VAR')
v.set_default('name', 'default_val')

ap = argparse.ArgumentParser()
ap.add_argument('--arg', type=str)
args = vars(ap.parse_args(['--arg', 'arg_val']))

v.bind_args(args)

print(v.get('name'))
os.environ['ENV_VAR'] = 'env_val'
print(v.get('name'))

# Result
#   default_val
#   env_val
# Expected result
#   arg_val
#   arg_val
#   Because Overrides > Args > Env variables > Config files > Defaults

Even worst, the Argparse binding mess up the default parameter
import vyper
import argparse

v = vyper.Vyper()
v.automatic_env()

ap = argparse.ArgumentParser()
ap.add_argument('--arg', type=str)

v.register_alias('arg', 'name')
v.set_default('name', 'default_val')
v.bind_args(ap)

print(v.get('name'))

# Result
#   None
# Expected Result
#   default_val

Indeed, the changes I did to the binding of arguments, may have had some unintended consequences. I'll take a look at the first issue ASAP.
The second issue you point out is "normal" though because you didn't specify a default when you added the argumentap.add_argument('--arg', type=str). If you set the default after bind_args() it should work as intended. I had already added the following to the README.md explaining the change in behaviour of bind_args() in relation to defaults:

The method also sets defaults based on what you pass via add_argument() default parameter.
Note: If you don't specify a default, the values will be set to None.

It was not in the release notes though, while it should've been.
Also, while personally in my work/projects I do use arguments as the defaults, maybe it should not be the default behaviour of bind_args() and/or maybe it should have a parameter to disable/enable that behaviour.

Thanks for your answer ! It s quite difficult to anticipate the behavior since a lot of things depends on the order in the vyper initialization. Another example of this is:
import vyper
import os

v = vyper.Vyper()

os.environ["TEMP_VAR"] = "original"
v.automatic_env()
v.bind_env("test", "TEMP_VAR")
assert "original" == v.get("test")
v.set("test", "modified")
assert "modified" == v.get("test")
os.environ["TEMP_VAR"] = "updated"
assert "updated" == v.get("test")

# Traceback (most recent call last):
#   File "/home/jbarbou/git_local/slack/bots/rodrigo/rodrigo/test.py", line 14, in <module>
#     assert "updated" == v.get("test")
# AssertionError

Seems that the register_alias is still under construction :-) the quotes between the "alias"shouldn't be there I guess.
IMHO, it could be good to flatten the config before returning the values as it is done in the go lib but I'm not an expert in go so I might get the wrong idea.
Anyway thanks again !

Seems that the register_alias is still under construction :-) the quotes between the "alias"shouldn't be there I guess.

Yeah, that should've been there, don't know why it was like that, but I fixed that in 67d1507d1c953357f1e5912b603ecaf99228e3c5.
I am working on the other issues.

Thanks for looking into this, it is a big tricky but the debug function helps a lot !


Add support for PAR1

PAR1 is a file format which uses a Reed-Solomon code similar
to the current one, except it uses a different (flawed) coding
matrix.
Add support for it via a WithPAR1Matrix option, so that this code
can be used to encode/decode PAR1 files. Also add the option to
existing tests, and add a test demonstrating the flaw in PAR1's
coding matrix.
Also fix an mistakenly inverted test in testOpts().
Incidentally, PAR1 is obsoleted by PAR2, which uses GF(2^16)
and tries to fix the flaw in the coding matrix; however, PAR2's
coding matrix is still flawed! The real solution is to build the
coding matrix like in this repository.
PAR1 spec:
http://parchive.sourceforge.net/docs/specifications/parity-volume-spec-1.0/article-spec.html
Paper describing the (flawed) Reed-Solomon code used by PAR1:
http://web.eecs.utk.edu/~plank/plank/papers/CS-96-332.html

Thank you for the great contribution. Everything LGTM. 👍


Website setup

The current website setup needs to be more straightforward and has many unknowns for new users.

How is it built?
Why do I need this tmp folder?
Where is the rest of the page stored?

Overall the initial setup is fine, but it adds a lot of boilerplate for building it, and pull request verification is also challenging.
Goal
We want to make the Developer Experience for the documentation smoother. Ideally, we separate the content from the layout. And reduce the obvious boilerplate for developers.
Proposal
As Hugo is powerful, we can leverage a lot of Hugo functionality to achieve a leaner structure.

we make the documentation build within this repository with Hugo features, instead of a git clone we will include the layout with Hugomodules.
module:
  hugoVersion:
    extended: true
  imports:
  - path: github.com/aepfli/lifecycle-toolkit-docs #current fork we need to adapt the go.mod file to have the correct name :)
    mounts:
    - source: static
      target: static
    - source: layouts
      target: layouts
    - source: data
      target: data
    - source: assets
      target: assets
    - source: archetypes
      target: archetypes

[x] adapt go.mod file in layout repo
[x] remove git clone from make file
[x] add config.yaml to doc directory and a proper go.mod file (https://gohugo.io/getting-started/configuration/#merge-configuration-from-themes important for proper merging)

use the outcome of the first one to generate an integrated build pipeline within the this repository

[ ] neflify build with preview
[ ] pull request link verification with markdown-link-check and htmltest
[ ] ensure that we do have a robot.txt which ensures that this page is not getting indexed
[ ] link the dev version to this netlify build. Due to that, we don't need to copy it

improve the main site build

[ ] separate content and layout (two distinct repositories) (this way we can use the layout even for other pages)
[ ] link the dev version in the dropdown to the main branch build of this repository
[ ] ensure that the docs upload is working as it is without the dev part

what do we solve with this?
The build is self-contained even in this repository.
We can easily verify that our current version is in a proper status without invoking another repository or build.
Our adaptions within this repository are still relatively low (3 additional files (go.mod, go.sum, config.yaml) and reduce makefile logic.
Additionally, we reduce the need to copy/paste the main branch.
Discuss

do we want to separate the layout and content?
does this approach sound desirable?

Sounds like a good approach, I like it!

I also like the approach since it:

allows custom local build
reduce copy/paste
simplify pipelines
keeps the docs next to the code

Great suggestion @aepfli 🚀

this is something, which might now be a little bit controversial, but I'll put it here anyways :) it is just a brain dump :)
Do we need our dedicated documentation repository for the content? eg. Jekyll and github_pages have their pages branch where they keep the content.
Pros:

it would reduce the complexity of pushing to another repository
we would have one netlify build for everything. We would need to set the default netlify branch to the page branch.
all changes to already done documentation will take place within the same repository - which generates IMHO a better overview

Cons:

one additional long-living branch
still not 100% clear regarding build/etc

I think i discussed that with someone already 🤔
I don't like that approach cus then you have a suuuuuper divergent long living branch in the same repo with the code. I'd rather have that separated. That's my only problem with it.
Other than that, it would make things a little easier if we had everything in the main repo, but then we have a versioning issue again potentially, with all the code for all version always sitting there.

We two talked about this. I would only put the webpage into this branch - no other source code, just the page. In our initial discussion, we discussed a long-living branch per minor version, which would be a lot to maintain. This way, the documentation is the only content in there. and instead of copying it to another repository, we would write it to the same repo in another branch.
This was a fancy idea of reducing the number of repositories and keeping the source of truth within one.

Theory crafting
We do have our dev docs within the main branch, and we do have a pages branch as our serving branch for all versions of documentation.
release workflow
As soon as a new release is created, the docs folder of this release is moved to the pages branch and gets suffixed with the version.
Additionally the version dropdown gets enhanced with the new version.
If the release is tagged as latest release:

we will move the existing docs folder to do docs-{version}. The version is determined by a version file within the docs folder, or as part of the frontmatter of the _index.md within this docs folder

we will use the existing docs folder of the main branch, copy it to the pages branch. adapt the frontmatter and create a version file containting the version

change the version menu generator to look for all docs folders and add them based on their version file, but .

just the last 5 versions?
or the last supported versions?

Following snippet would be a version menu created based on folders, rather than configuration

docs/layouts/partials/navbar-version-selector.html

	{{ .Site.Params.version_menu }}

	{{ range sort .Site.Pages ".File.ContentBaseName" "desc" -}}
      {{ if (findRE "docs" .File.ContentBaseName) -}}
      {{ $version := replaceRE "docs-" "" .File.ContentBaseName -}}
      {{ $version }}
      {{ end -}}
      {{ end -}}

3 unrefined versions of building a verison menu:


Add support for mysql 8.0.3+

I am currently running

SpongeForge version: 1.12.2-2705-7.1.0-BETA-3206
Java version: HotSpot 1.8.0_131-b11 64-bit
Operating System: Ubuntu Xenial
Plugins/Mods: none

Issue Description
When using sponge database service with mysql 8.0.11, db initialization in my plugin crashed with exception:
com.zaxxer.hikari.pool.HikariPool$PoolInitializationException: Failed to initialize pool: (conn:10) Unknown system variable 'tx_isolation'

Code example:
SqlService service = Sponge.getServiceManager().provide(SqlService.class).get();
String aliasKey = conf.getNode("db", "mysql", "spongeSqlAlias").getString();
db = service.getDataSource(service.getConnectionUrlFromAlias(aliasKey).orElse(defaultDsn));

This issue related to maria db jdbc driver and was resolved in release 2.2.5. Corresponding ticket: CONJ-604.
Solution
Update mariadb-java-client dependency version to 2.2.5.
Incompatibilities
Not found. According to MariaDB Connector/J release notes from 2.0.3 up to 2.2.5 there is no BC breaking changes.
Possible workarounds
Clone sponge repo, change dependency version and build. Or repack existing sponge jar with new mariadb driver. This is bad and not recommended.

@gabizou What amount of testing would be required to get this relatively simple dependency change into the next minor/major (assuming has binary changes, and that the connector follows semver) SpongeAPI builds.

I've done a minimal amount of testing against org.mariadb.jdbc:mariadb-java-client:2.3.0 .
Using that connector you can indeed make a connection, and execute a query.
I'm not expert of the advanced usages of jdbc connectors, so I'm not sure what else would need testing, other then running with some plugins that use jdbc connections.

Okay, my apologies for not acting upon this all that time ago.
There was another issue whereby the default MySQL 8.0.11+ authentication mode wasn't supported and from what I see, that was resolved in version 2.5. So we're going to see if the 2.6 release will work with what we expose and, if so, we'll push that out.

Please fix this already. Literally all it takes is change 1 line.
I've been using versions of the MariaDB driver that supported MySQL 8 since the beginning in my plugins and I've yet to come across any issue whatsoever.

Currently looking at writing some regression tests for this upgrade. Could someone detail what exactly you would want to see tested before making the change?

After investigation, we're just going to bump it. It should work fine in all but corner cases, but as this is in impl for the most part, it should be fine. We'll just make a big point of the upgrade on our release notes.


[T4A2][W13-B3] Mok Wei Xiong, Edmund

Accepting this PR without a review because another of your PRs for this tutorial was reviewed in detail.


Green Fast Track support

Project Description
DE will extract source data, do geospatial calculations, and produce outputs needed for the new Green Fast Track survey GIS is making.
Timeline
Friday 2/23
For a demo of the POC, DE and GIS will focus on a subset of CEQR categories: Zoning, E-designation, Air, Noise.

[x] https://github.com/NYCPlanning/data-engineering/issues/576
[x] https://github.com/NYCPlanning/data-engineering/issues/575
[x] Add build logic

https://github.com/NYCPlanning/data-engineering/issues/609
https://github.com/NYCPlanning/data-engineering/issues/607
https://github.com/NYCPlanning/data-engineering/issues/608

[x] Export FileGeodatabase for use in survey

https://github.com/NYCPlanning/data-engineering/issues/590

[x] Show the Green Fast Track POC

Wednesday 3/20
Planning Support will host a forum with applicant reps for feedback on Eligibility Tool. The goal is to confirm it would work for their workflow.

[x] By 3/6 share survey language with Planning Support
[ ] NYCPlanning/edm-overview#1029
[x] On 3/11 Review Tool with Planning Support and EARD
[ ] #664
[ ] https://github.com/NYCPlanning/edm-overview/issues/1012
[ ] By 3/14 discuss outline of Tool demo and get outputs for pilot projects
[ ] Demo at Applicant Forum for ~10 minutes

Background
The City Environmental Quality Review (CEQR) process identifies and assesses the potential environmental impacts of land use actions that are proposed by public or private applicants.
DCP plans to streamline housing construction by allowing potential applicants for land use actions to determine if their project is minor enough, known as CEQR Type II, and therefore exempt from environmental review. This may become known as the "Green Fast Track Eligibility Tool".
GIS is creating a survery for potential applicants to use and determine if their project is CEQR Type II .
The determination will be based on the project area.  For each tax lot, all relevant CEQR considerations must be checked.

Potential geospatial logic
Zoning districts

wholly within an existing R5 through R10 Residential zoning district.  Logic: The 4 zoning district fields must only include R5 through R10 values and be absent of any other zoning district.
partially within an existing R1 through R4 Residential zoning district.  Logic: An R1 through R4 value appears in 1 of the 4 zoning district fields
within in a Special Coastal Risk District.  Logic: One of the 3 special purpose district fields contains a CR% value. CR - n translates to Special Coastal Risk District, where n is the number of the district.

Air quality
Add three binary fields indicating if the tax lot intersects with any of the following buffers:

400 Feet of an active or expired CATS permit.  Data Source: https://www.ceqr.app/data/air-quality
1,000 Feet of a State Facility Permit Source.  Data Source: https://www.ceqr.app/data/air-quality
1,000 Feet of a Clean Air Act Title V Permit Source.  Data Source: https://www.ceqr.app/data/air-quality
The dataset are already being ingested in workflows for the CEQR Air.  We'd need to confirm that: 1) the CATS permit data that we're pulling in contains expired permits and 2) how often will we have EARD manually review and geocode the data, which they currently do with each update.

Arterial highways and vent structures

Buffer arterial highways (lines) and vent structures (polygons) by 75 feet and flag if a lot intersects with the buffer.
Arterial highways data source: Appendix H zip file in project folder from TRD.  Question for GIS and TRD: can we make this data part of an existing regular data exchange?
Vent tower data source BBLs:<PHONE_NUMBER>,<PHONE_NUMBER>,<PHONE_NUMBER>,<PHONE_NUMBER>,<PHONE_NUMBER>,<PHONE_NUMBER>,<PHONE_NUMBER>,<PHONE_NUMBER>  - Question for PS: How was the vent tower list created and how will it be maintained?  Question for GIS:  Is there a better data source?

Elevated subway or railway

Logic: if a lot intersects a 1,500 foot buffer of an open cut, at-grade, and elevated NYC Subway, SIR, MNR, LIRR, AirTrain and other freight lines it gets flagged.
Data source: TBD.  LION?  Or manually created file from PS?

Airport

Logic: A lot intersects the 2021 65 DNL Contour for LGA or JFK airports.
Data source: TBD.  Need to get data from contact and establish update procedure

Natural Resources and Shadows

Logic: A lot intersects a 100 foot buffer of the following natural resources
State Regulated Freshwater Wetlands and Checkzone: https://data.gis.ny.gov/maps/a57e144caedb4b1aaf510809013e4ac7/about
Wetlands that are currently mapped or officially proposed for addition to the wetland maps and currently regulated under the NYS Freshwater Wetlands Act.
Tidal Wetlands: https://data.gis.ny.gov/datasets/661acb5eaffb4be39b0d6d2203e636c3_1/explore?location=40.889803%2C-73.055095%2C9.87
Priority Waterbodies: https://data.gis.ny.gov/maps/fe6e369f89444618920a5b49f603e34a/about
Lakes, Estuaries(includes Rivers), Streams, Shorelines
Significant Natural Communities: https://data.gis.ny.gov/datasets/0da09cdf37d549b1be9add9b522ee505_0/explore?location=40.601737%2C-73.998364%2C11.83
Beaches: Need to determine data source.  Is it the parks properties layer or encompassed in another existing natural resource dataset?
National Wetlands Data: https://fwsprimary.wim.usgs.gov/wetlands/apps/wetlands-mapper/
I think there is a code in here for land mass, we would want that removed
Question for PS:  Does someone need to know what a lot intersects with, or can all of the above be grouped into one flag?

Historic and Shadows

Project contains a historic resource
Project is within 90 feet of a historic resource
Data sources:
LPC NYC Individual Landmarks, Historic Districts, and Scenic Landmarks: PLUTO historic district and landmark status fields OR https://data.cityofnewyork.us/Housing-Development/Individual-Landmark-and-Historic-District-Building/7mgd-s57w and https://data.cityofnewyork.us/Housing-Development/Scenic-Landmarks-Map-/gi7d-8gt5
NYS OPRHP National Register Historic: https://data.ny.gov/Recreation/National-Register-of-Historic-Places-Map/y36f-mkpp
Need to contact them to ask what the appropriate data to use is.
NYS OPRHP State Register Historic: TBD
Need to contact them to ask what the appropriate data to use is.

Open Space/Shadows

Logic: If a tax lot touches a 75 foot buffer of the following
NYC Parks properties https://data.cityofnewyork.us/Recreation/Parks-Properties-Map/krz2-j7bn
Remove parking lots and parkways (roads) from data
NY State Parks Property | NY State Parks Property | NYS GIS Clearinghouse: https://data.gis.ny.gov/datasets/nysparks::ny-state-parks-property/explore?location=40.843794%2C-73.902887%2C10.95
POPS: https://data.cityofnewyork.us/City-Government/Privately-Owned-Public-Spaces-POPS-Map/qeta-4kqg#:~:text=Privately owned public spaces%2C also known by the,into New York City’s zoning regulations in 1961.
PS would prefer to have a polygon dataset.  Can we get POPs polygons?

E-designations

Report what E-designations are on the tax lot
Need to determine if there is any logic beyond this (i.e. joining to e-designation table to get type of e-designation)

Just wanted to clarify - who is point person (on our end) for questions for PS?

"our end" meaning GDE

@croswell81 and @jackrosacker please see my answers below

Zoning districts (Zoning Values within PLUTO) • Is one zoning district range being assigned to a single column or are we checking off if those zoning ranges apply to the project in three Boolean columns? • Is there a hierarchy for the different zoning district ranges: R1 through R4, R5 through R10, C or M? (i.e. if a BBL falls into multiple categories is one range assigned over the others)

If it's possible to calculate the Zoning District for the CEQR II form using the existing 4 zoning district fields in PLUTO that may be preferable.
To be considered for the R5 through R10 Residential zoning district the 4 zoning district fields must only include R5 through R10 values and be absent of any other zoning district.  Whereas to be considered as an R1 through R4, an R1 through R4 value must simply appear in 1 of the 4 zoning district fields.  A BBL is assigned a zoning district if 10% or more of the bbl's area is covered by that zoning district.  Zoning district values are assigned from most covered to least coverage, with Zoning District 1 covering the greatest area and Zoning District 4 covering the least area.  For CEQR II we just need to know if a project is R5 through R10 or R1 through R4.

Arterial highways and vent structures – GIS Team needs to review the sources
• Arterial highways: if a project is located next to multiple features, do we need to list the name of all or just one? Closest?
We just need to know if a project is near an arterial highways or vent structure, or not.  This would be a boolean value; we do not need to report any name.
• Arterial highway source – is this just the arterial highways in the DCM_ArterialsMajorStreets open dataset
That is a question for Andrew E.
• Vent tower: is this a distinct question or part of the arterial highway question? Do we need to list the name of the vent?
You do not need to list the name of the vent.  Check with Planning Support if vents need to be reported separately from highways.

Elevated subway or railway • if a project is located next to multiple features, do we need to list the name of all or just one? Closest? • Source: GIS Team to compare LION vs data received from PS
We just need to know if a project is near an elevated subway or rail, or not.  This would be a boolean value; we do not need to report any name.

Natural Resources and Shadows – GIS Team needs to review the sources (who determines what source is valid?) • if a project is located next to multiple features, do we need to list the name of all or just one? Closest? • Noticed a state and federal wetland dataset – should we use only one? How do we handle conflicts? • Beaches: need to identify a source
Like the above, we just need to know if a project is near a natural resource and not what it is.  Given that we just need to know if something is near something we don't need to worry about conflicts.  I proposed GIS team recommend what the data source should be used.

Historic and Shadows – GIS Team needs to review the sources • if a project contains or is located next to multiple features, do we need to list the name of all or just one? Closest?
Same as above.

Open Space/Shadows – GIS Team needs to review the sources
• Should we include federal park properties?  I'd ask PS.

E-designations • We need to create a field for each type of e-designation (noise, air quality, hazmat)? • For lots with multiple e-designations, should they be concatenated into one field? • Should we include restrictive declarations (i.e. e-numbers that start with “R”) • Source: e-designation table (csv)
We need to work through e designations with PS and EARD.

Regarding GDE POC to PS and EARD - I'd prefer to not play telephone, so whoever is doing the work should reach out directly, but loop me in on all communications and come to me first if there are any questions you have that you want to discuss internally before reaching out to PS and EARD.

from 1/29 sprint planning

awaiting confirmations of product specs and data sources
in the meantime, would be nice to draft the build architecture for this new product

expected to build monthly
will not depend on PLUTO data
but if the spine of this new product is identical to PLUTO, maybe it should be part of the PLUTO build

I am cleaning up the CEQR_Type_II_Data_Source_Review.xlsx doc. I will let you know when all edits are completed.

We've imported all source data and confirmed output specifications for the POC
Now working on build logic. To help keep GIS unblocked, we'll export mock data by tomorrow morning.

notes from Survey demo on 2/23

nice map
would prefer auto-population of survey

requires changing radial buttons to text
difficult to pass along values from multiple BBL records

want to know values of some adjacent features, not just the nearest (e.g. all CATS permits, but not all elevated railways)
have to address applicant flexibility

e.g. line of site vs within 1,500 feet of elevated railway
allowing applicants to challenge an auto-answerer question as part of their submission
some survey questions cannot be addressed with data. some can only be answered with user data

2/26 meeting
general

we'd like the Survey be paged to group separate the two sets of questions: questions DCP answers and questions the Applicant answers

e.g. historic resources, line of sight

nothing is confirmed until Applicant finishes survey, exports, and interacts with DCP
there are three phases: Spatial, XXX, Proposal
readability is higher priority than machine readable (e.g. comma separated permit IDs vs long table)

BBL and address

we want to prioritize getting BBLs from applicants

Zoning Districts

There are 3 categories: (1) partially within R1-R4, (2) wholly within R5-R10, (3) wholly within C or M
Survey can't do aggregations/logic to determine "your project is wholly within an M based on your three BBLs"
A user would input whether: they're 1 or 2 and whether they're partially or wholly within 3
For MX districts, projects cannot change MX

Therefore, they must be informed of what "C or M" they're in so they can know they're in an MX

These are two questions

"what R category are you in"
"how much of the project is in a C or M"

Special Coastal Risk District

binary: "project is partially/wholly in one"
we should include/export the offending value (for each lot)

Air Quality

for spatial questions, what does DCP need to know? if a single BBL is within XXX feet of a CATS permit?

need to know the ID adjacent to each BBL

are you within 400 feet of a manufacturing land use?

"industrial land use screening"
binary answer is the priority
the BBLs adjacent to each BBLs is valuable but is a nice-to-have, a count may work too

Noting a couple of data items that popped up last week that I don't want to forget:

Arterial Highways will need to have the 75' buffer applied from road edge, not centerline. This could likely be done by using the centerline data to select and then buffer planimetric data. Rodrigo or Matt are probably the best resources for understanding the planimetric dataset - might be a flag or attribute to select relevant features without using the centerline.
It looks like we will need to merge/dissolve all buffers per category for cartographic purposes. I ended up doing this manually for the demo because I couldn't get the multi-buffers to work visually. This results in lost resolution in terms of what a user can see via popup, but I think that's ok.

Since Arterial Highway is such a specific dataset, I would recommend keeping DCM arterial centerline data and doubling the buffer to 150 feet. We should confirm with Planning Support and EARD but it would be a much easier task than trying to recreate the arterial highway list from another source.
@jackrosacker

Could we use the centerline to select associated planimetric features within a set distance, and then base the 75' buffers off of those selected features? (Damon's idea from last week I think)

It would take someone going in and cleaning up all other roadways that are within that 75 buffer. The planimetric data doesn't have attribute data to narrow down what could be selected.

Also, we should remember this is a flag, not a precise calculation. Using the Arterial Highway is the most accurate dataset, that allows us to use an existing public resource. Adjusting the buffer is the better way to go.
The planimetrics dataset is derived from aerial images and is updated every 2-4 years, not as reliable.

Okie doke. We can run the increased buffer idea by folks in the pm meeting today

@croswell81 @jackrosacker
During one of our chats with Planning Support, there was mention of an Air Quality check I don't think we're doing yet: "are you within 400 feet of a manufacturing land use?"
It seems like the logic for this flag would be something like:

For a single project lot, find all lots within 400 ft
For all of those adjacent lots, are any of them manufacturing?

a lot is manufacturing if any of it's zoning district values start with M

If so, flag as True and list the BBLs

And the source data to pass along for the map would be "all manufacturing lots"
Does that sound right?

That sounds tentatively correct to me, pending any changes from Matt.I added a placeholder question for this in the survey as well, until we have final language. I forwarded you the email from Alex with the initial question, and placing the email text here:
Hi ITD Team,

We had a discussion with Stephanie today that included the desire for an addition to the air quality section of the tool. We currently do not have any logic to flag the potential for an unpermitted air quality source, and it is just in a note necessitating a manual check. We are hoping it may be possible to query within a 400 foot buffer from the site if there is any lot with a manufacturing land use (using PLUTO). If there is, we would flag that they need to check this study area for unpermitted industrial sources.

The question would be something like: Are there any manufacturing or processing facilities operating within 400 feet of the Development Site that may be unpermitted sources of air quality emissions?

We are happy to use some of our time next week to discuss.

Definitely an item to work on further with them.

@damonmcc @jackrosacker They are looking to identify uses, not zoning, so the query or filter should use Land Use code = 06.
We should confirm with Planning Support and EARD there are no other Land Uses or building codes that allow "processing facilities" besides '06'.

@damonmcc et al, I spotted a couple of possible issues with the zoning classifications within the green fast track bbl dataset.
Item 1: Lots with some R1-R4 present are getting classified as R5-R10

I this example, lot 1 (red underline) is classified as R1-R4, while lot 70 (blue underline) is classified as R5-R10. I think both should be R1-R4, since any part of the lot in both cases is within that range. I could be wrong about this, so let's discuss before you start re-engineering anything.
Item 2: Lots with no zoning category at all
See BBL<PHONE_NUMBER>
I haven't done any form of comprehensive error checking yet, just flagging these as they pop up

From my understanding, we (AD and PS) agreed on "pluto classification" as truth for whether R1-R4 (or containing any specific district). That threshold is 10% of the district.

@jackrosacker
@fvankrieken beat me to it. We don't assign zoning to PLUTO lots if it less than 10% and that will apply to this app and data as well.

I'll look into<PHONE_NUMBER>

Ok, I'd lost track of the fact that we're following the 10% rule for GFT as well. I'll add that to the zoning aggregation description in my presentation.
@fvankrieken - seeing ~10k lots without any zoning classification in what I believe is the latest version of the dataset. There might be a legitimate reason for some of these that's eluding me

4124950002 is based on the original zoning logic, which said one of

wholly R5-R10
partially R1-R4
wholly C/M

4124950002 is R6 and C4, meaning it fits none of those three categories. I think you and @damonmcc had come up with a revised little decision tree, which will need to be implemented.
Though still not quite sure what this should be "flagged" as? For this specific case of R6 and C4?

Pretty sure this one would be a C or M lot. My understanding of the language is that it only requires the presence of C or M, not necessarily "wholly."
This is probably a post-demo discussion. Sounds like we should discuss and refine the zoning decision tree, and choose a non-null text value to indicate lots that are in none of the buckets above ("Other", "Ineligible", etc.)
As I'm writing this I'm wondering if we also need to account differently for lots that are a mixture of R1-R4 and C or M.

After the demo, maybe GIS and DE can get together again to review the logic of each field, including updating buffer distances.

Logging some thoughts here for how to account for lots with both R1-R5 and C or M present. Each option represents a hypothetical project site in which four lots are selected, each with the following zoning values.
Option 1: Table has a single zoning category column, with four possible options per BBL.

BBL
Zoning Category

1111111111
R1-R4

2222222222
R5-R10

3333333333
C or M

4444444444
R1-R4 with  C or M

Option 2: Table has two zoning columns, resi and c or m. There are still four possible zoning options per BBL, but pulled from values combined across both columns. This example uses the actual value within the C or M column, not just a Yes/No option.

BBL
Residential Zoning
C or M Zoning

1111111111
R1-R4

2222222222
R1-R4
C or M

3333333333
R5-R10

4444444444

C or M

Option 2b: Table has two zoning columns, resi and c or m. There are still four possible zoning options per BBL, but pulled from values combined across both columns. This example uses a Yes/No value within the C or M column, not the actual value.

BBL
Residential Zoning
C or M Zoning

1111111111
R1-R4
No

2222222222
R1-R4
Yes

3333333333
R5-R10
No

4444444444

Yes

For our upcoming GFT data meeting @croswell81 @damonmcc @fvankrieken (since I think you're tackling the zoning stuff?)

@damonmcc @jackrosacker They are looking to identify uses, not zoning, so the query or filter should use Land Use code = 06.
We should confirm with Planning Support and EARD there are no other Land Uses or building codes that allow "processing facilities" besides '06'.

@damonmcc I'm bringing unpermitted industrial lots into the GFT app. Planning to do it a la (e) designations, in which I filter the green_fast_track_bbl layer down to the relevant lots. Can you confirm the logic that you ended up using to calculate this dataset on your end? Was it the same as Matt presented above, or has it changed since then?
Side note - added a corresponding row to our data source sheet that we can fill in together at some point.

@jackrosacker should Option 2 have more specific values in the C or M Zoning column? it says "This example uses the actual value within the C or M column, not just a Yes/No option."

notes from DE & GIS chat on 4/2
Lot Zoning info

R1-R4 can exist with C or M
R5-R10 cannot exist with C or M
for current NULL values use one of these: Ineligible, Not Applicable, Other

Natural Resources

survey questions:

does your porject site contain a natural resources?
is your project site near a wetland check zone?
Are you near another ... ? (not data-driven)

App Table vs Output CSV

show single binary Natural Resources answer
show all columns related to that answer

Historic Resources (Alex)

distinction b/w resource and district

resource = point
district = polygon

"does your project contain" (look for points)
"are you near" (buffer lots that contain points or buffer points)

Rail

GRU is gonna make changes for 24B

Rail Yards

GIS can use polygons (Complex Polygons)
not in CSCL because they're Complexes

Beaches

Non-DPR parks have no names

@jackrosacker @croswell81
from our chat about Lot Zoning vs Project Zoning, I tried to illustrate the logic we'd implement in order to do Option 1 above (one Zoning column). I'll put the diagram in this comment and this link to the PR it's coming from
Logic for Lot Zoning in GFT

@damonmcc @jackrosacker This looks correct to me.

Agreed, with one amendment: Any R5 - R10? should be Entirely R5 - R10?
@damonmcc @croswell81

thanks @jackrosacker! I think I see what you mean about changing to Entirely, so I revised it to be the diagram below.
I wanted to still use Any because that frame of mind translates really well to the logic/code we'll have to write. Hope this captures it!

Cool, yeah that seems to cover more bases. I wonder if there's ever a circumstance in which there's a third branching option from the Any R5-R10 in which the lot is partially Other, meaning that the lot could not be classified as "Entirely R5 - R10". Or other possible edge cases? I'll leave subsequent comments in the #741 issue

@damonmcc I wanted to also suggest that we find and record a BBL for each possible zoning combination to use as benchmarks. If that's easy for DE to gather while writing the queries, could you add a list of BBLs to this issue? I'm happy to poke around and find them as well if helpful.

@damonmcc I've been able to successfully publish the source_data_versions table to ArcGIS Online, and pull the dataset name and vers/date values into the survey as static values. This will enable us to print out the data versions at the bottom of each report PDF for easy reference. A few things occurred to me while implementing this:

Are the column headers finalized for this table? The naming convention doesn't matter at all, but the report will break if we change convention later
Noticing variability in the date format in the 'v' column. I personally prefer the 1900-01-01 format. Requesting that we normalize to one format
Are we confident that each of the version/date values are getting updated when each build happens? I don't see anything to indicate otherwise, but want to be sure these values are accurate before printing into the report
Do the dates indicate a specific thing? i.e. date the data was ingested for external, date the data was built for internal, etc.

This doesn't have to happen before the Beta.

Are the column headers finalized for this table? The naming convention doesn't matter at all, but the report will break if we change convention later

The column headers are finalized. This table is generated by code that we use in all builds to load source data.

Noticing variability in the date format in the 'v' column. I personally prefer the 1900-01-01 format. Requesting that we normalize to one format

Definitely possible. Since these are versions of source data, DE kind of treats these more like strings than dates. So changing the format sort of breaks the 100% certainty that we'll find that exact value in edm-recipes, but no worries! we're talking about formatting to display and DE can always retrieve the pre-formatted value if we need to.

Are we confident that each of the version/date values are getting updated when each build happens? I don't see anything to indicate otherwise, but want to be sure these values are accurate before printing into the report

Yup this file green_fast_track/recipe.yml ensures that every build uses a particular version (in this case the latest version) which is then documented in the source data version table.

Do the dates indicate a specific thing? i.e. date the data was ingested for external, date the data was built for internal, etc.

This varies by dataset, specifically the source of the dataset. If the created data is programmatically available, the version is the created date. If it isn't available, the version is the ingested date.
Everything we ingest from NYC Open Data or an ArcGIS feature server uses the "last updated" value as the version.
We'd like to make this more clear for all source data and perhaps could reflect the details by one or more new columns in this table, but don't plan to do that soon. We may be able to describe what the version means for each dataset in DE's GFT documentation though!

@damonmcc thanks for this. Glad to hear that the data is reliably up to date, and that the field names aren't expected to change. Based on what you're saying, I think we should:

leave the date formats alone, unless there's another reason as well to align them
add a column to indicate what the date means for each dataset - if this needs to happen down the road in the table that the build outputs, we can manually enter values for each row in the GH data sources page and on the table at the end of the PDF report (Casey built a solid first draft of this, we can share with you next week!)

potential source data changes/additions

Exposed Rail Yards

confirming how to use CSCL

Beaches

no need for names, may just list municipality (city, state, federal)

Ecological Complexes
Unpermitted Industrial Lots

will decide on lots vs zoning

Groundwater

need DOB BIS data

DPR changes

waiting for confirmation on filter we use for DPR park properties

from @jackrosacker on 5/1
priorities re DE items for the Beta launch on 5/3
Priority 1:

bbl dataset with new zoning column and per question flag columns
DE review of my assumptions in column D of the Data Source sheet, OR, you and I finalizing the new sheets we've been working on to verify any assumptions in matching datasets
[done] Add missing historic resource lots
Priority 2:
Shadow/Historic Resource buffer (I'm going to mock this up in the meantime so I have a dataset to visualize, can swap out the data source when this is ready from Alex)
Priority 3:
DE query out all successfully lot-joined points (potential - I'm still playing with whether I can cleanly do this on our end)

@fvankrieken and @alexrichey, following up on our conversations this week with a punch list of Fast Tracker items. I've tried to be as comprehensive as possible, but there are a few items such as dataset aggregation and column/dataset naming review that will probably benefit from a closer look together before DE puts in too much work.
Also - this list alludes to but does not fully cover the remaining datasets to be added. These fall into two buckets: (1) datasets that have been cleared by Matt/Planning Support and handed off to DE but haven't been processed yet, and (2) datasets that are still being digitized/approved.
I'm out of office this afternoon but could talk through and help prioritize the below list tomorrow if either/both of you are available.

Create new Historic Resource buffer (in progress)

Incorporate all outstanding datasets that have been made available by Matt etc. (e.g. rail yards, DOB wetlands) - Matt will have more detailed information on these

Update dataset names for consistency
a. No huge changes needed, just to review conventions across entire population of datasets and ensure rough consistency. Happy to do this review together.

Update arterial hwy buffers - still using the old distance. Review all other buffer distances for other possible

Create a data sources page in Github - beta is currently linking straight to root of DE GH

Update green_fast_track_bbl dataset:

Finish adding per-question flag columns

Existing, in 4/25 GDB:

State_Regulated_Freshwater_Wetlands___Checkzone_Flag
State_Archaeological_Areas_Flag
Natural_Resource_Shadow_Flag

Needed, calculated by GIS Team from 4/25 GDB:

Natural_Resource_Flag
Historic_Districts_Flag
Historic_Resource_Flag
Historic_Resource_Adjacent_Flag
Open_Space_Shadow_Flag
Historic_Resource_Shadow_Flag

Add Natural Heritage Communities name/id column

(optional) Ensure that any outstanding datasets have a corresponding column created, even if all values are null)

Edit field names for consistency:

Name character count <= 25 characters (see existing, some are as long as 52 char)
Leading character must be alphabetical (see existing, some with leading '_' char)
Maintain naming conventions across the entire dataset: i.e. If multiple admin levels of a dataset are included (i.e. historic resources) make sure that each corresponding field explicitly names that level (i.e. nyc, nyc, us)
Aliases don't matter for now - I re-alias anyway on my end

Take note of any fields that keep the name but change significance. See Damon's example re Historic Districts: "And there's one flag that will be renamed (and not show in the tool anyway) as to not conflict with a question flag: Historic Districts -> City Historic Districts."

Spelling of LaGuardia in airports dataset -> currently all lowercase

Return all point datasets as filtered to include only those points that have not been successfully joined to a lot (this is already the case for, say, historic resource pts, but not for CATS Permits)

Create single dataset per question and per geometry type for the latter sections: Natural/Historic/Shadows

For buffers: merge and dissolve all constituent datasets into a single feature
For features: merge all constituent datasets and retain variable_type and variable_id values for each feature/row
Examples, using the datasets I manually produced (I used a "gft_" prefix, that doesn't have to remain static):

gft_shadow_open_spaces_buffer
gft_shadow_open_spaces_lots
gft_historic_resources_points
gft_historic_resources_lots
gft_historic_resources_buffer
gft_historic_districts
gft_natural_resources

(I have a more in depth table of these transformations and examples of the outputs produced, if helpful to review together)

@damonmcc @croswell81
I added data to our running GFT Data Sources sheet.

Added columns to the Survey Questions tab:

App Flag Alias -> these values are what I'm currently using in the app as aliases for the flag columns
Possible Flag Field Name -> these are possible green_fast_track_bbl field names, based on my aliases and your existing field names (my alias uses "elevated" because that's in the survey, but DE switched to "exposed" which I like)

Added a Flag to Name Pairing tab to capture the name/id aliases that I'm using, paired with the relevant flag. I haven't translated these into gft bbl field name ideas yet

Take a look when you get a chance and we can regroup as needed.

@damonmcc  @jackrosacker
Latest data updates (refer to bullet 2 in Jack's comment from 5/8 above). All data updates are reflected in the CEQR Type II Data Source Review doc.
Updates:

Zoning data: signed off (marked approved)
Exposed Railway (marked as approved)
a.	received updated version from GR
b.	can send via Teams or you can use latest LION and process - let me know what you prefer
Exposed Railyards (marked as approved)
a.	received queries from GR
b.	instructions are provided in the Data Processing and GDE Notes columns
c.     can send via Teams or you can use instructions to process - let me know what you prefer
d.	GIS will provide the Railyard_HudsonYard_erase.shp in Teams
Beaches (marked as approved)
a.	Updated the Name column with new values that should be returned in the CSV export table
b.	Will provide new data in Teams
NYCDOB Natural Resource Check Flags (marked as approved)
a.	GIS provided data in Teams – let me know any questions
b.	One spreadsheet contains flags for following:
i.	Tidal Wetland
ii.	Freshwater Wetland
iii.	Coastal Erosion Hazard Area
DPR Park Properties – No changes, keep as is, we never heard back from Parks

Pending:

Recognized Ecological Complexes (RECs) – working out methodology for this. Planning Support understands this will take more effort and OK it is not ready by Go Live. Will still try to get this ready for the 6/3 deadline.

[celebrate]     Jacob (Jack) Rosacker (DCP) reacted to your message:

From: Matthew Croswell @.>
Sent: Friday, May 10, 2024 4:27:54 PM
To: NYCPlanning/data-engineering @.>
Cc: Jacob (Jack) Rosacker (DCP) @.>; Mention @.>
Subject: [EXTERNAL] Re: [NYCPlanning/data-engineering] Green Fast Track support (Issue #486)
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.  Forward suspect email to<EMAIL_ADDRESS>as an attachment (Click the More button, then forward as attachment).
@damonmcchttps://github.com/damonmcc @jackrosackerhttps://github.com/jackrosacker
Latest data updates (refer to bullet 2 in Jack's comment from 5/8 above). All data updates are reflected in the CEQR Type II Data Source Review dochttps://nyco365.sharepoint.com/:x:/r/sites/NYCPLANNING/itd/edm/Shared Documents/PROJECTS/GIS/202401 CEQR Type II Survey Build/docs/CEQR_Type_II_Data_Source_Review.xlsx?d=wbae1537787a64188963736a89ac1df98&csf=1&web=1&e=LeHgTT.
Updates:

Zoning data: signed off (marked approved)
Exposed Railway (marked as approved)
a. received updated version from GR
b. can send via Teams or you can use latest LION and process - let me know what you prefer
Exposed Railyards (marked as approved)
a. received queries from GR
b. instructions are provided in the Data Processing and GDE Notes columns
c. can send via Teams or you can use instructions to process - let me know what you prefer
d. GIS will provide the Railyard_HudsonYard_erase.shp in Teams
Beaches (marked as approved)
a. Updated the Name column with new values that should be returned in the CSV export table
b. Will provide new data in Teams
NYCDOB Natural Resource Check Flags (marked as approved)
a. GIS provided data in Teams – let me know any questions
b. One spreadsheet contains flags for following:
i. Tidal Wetland
ii. Freshwater Wetland
iii. Coastal Erosion Hazard Area
DPR Park Properties – No changes, keep as is, we never heard back from Parks

Pending:

Recognized Ecological Complexes (RECs) – working out methodology for this. Planning Support understands this will take more effort and OK it is not ready by Go Live. Will still try to get this ready for the 6/3 deadline.

—
Reply to this email directly, view it on GitHubhttps://github.com/NYCPlanning/data-engineering/issues/486#issuecomment-2104894710, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AEKVJ2L7IZD7YM4I5IJCSCDZBTYQVAVCNFSM6AAAAABBOUHVPGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMBUHA4TINZRGA.
You are receiving this because you were mentioned.Message ID: @.***>

Noting to @damonmcc and @croswell81 that as I understand our design, the output CSV will have a single name/id column per dataset, meaning that e.g. if a lot both intersects with a tidal wetland (Nat Res question) and is also within 200ft of that tidal wetland (Shadow question), the name/id of that feature will appear once(?) in the single name/id column for that dataset, but will not differentiate in the export which question or flag it is associated to.
I think this makes sense to some degree, but had been vaguely assuming that the tidal wetland id would appear under a column for the natural resources intersection and another for the shadow buffer.
Does this line up with your understanding? Or am I missing something and we're planning to reflect that same tidal wetland ID in two columns, one per question/section?

@jackrosacker I was thinking that any resource that triggers a buffer that intersects a project lot would only be in one column (i.e. historic dist (contains, within 90 ft, within 200 ft - shadow) but realize the buffers are different and therefore all resources will not apply to all buffers and questions.
I think we should send to Planning Support and see if they care before we have DE add a bunch of new columns to the export table.
cc: @damonmcc

Started to plot this out in advance of emailing PS, and ran into a few other wrinkles. Let's take an example project like below:

Iteration 1 - Our current design would have a single CSV column per dataset, regardless of how many times that dataset relates to the project, or through which question/spatial relationship:

BBL
NYC Hist Res ID
NYS Hist Res Name

3030530013
F. J. Berlenbach   House

3030530016
F. J. Berlenbach   House

3030530019
F. J. Berlenbach   House

Iteration 2 - The alternative we discussed above, which ends up adding a column per question and per dataset, so that for each BBL you know the dataset, feature ID, and question/spatial relationship relevant:

BBL
NYC Hist Res ID
NYC Hist Res ID -   90'
NYC Hist Res ID -   200'
NYS Hist Res ID
NYS Hist Res ID -   90'
NYS Hist Res ID -   200'

3030530013
F. J. Berlenbach   House

3030530016

F. J. Berlenbach   House

3030530019

F. J. Berlenbach   House

Iteration 3 - A third option, in which each question has corresponding data name/id field in the csv, and names/ids are grouped with a categorical prefix to indicate the relevant data source (imaginary datasets in all caps to demonstrate how multiple datasets would be aggregated into a single column):

BBL
Hist Res ID
Hist Res ID - 90'
Hist Res ID - 200'

3030530013
NYC: F. J. Berlenbach House

3030530016

NYC: F. J. Berlenbach House, NYC: AN EXAMPLE RESOURCE, NYS: AN EXAMPLE RESOURCE

3030530019

NYC: F. J. Berlenbach House, NYC: AN EXAMPLE RESOURCE

Iteration 1 is the most concise, but makes it harder for an applicant or EARD to review and application and understand how the BBLS, flag datasets, and questions interact with one another.
Iteration 2 gives makes the review easier, but substantially increases the number of columns required.
Iteration 3 is a combination of the other two, with the benefit of having fewer output fields but data that is harder to parse per CSV cell.
After exploring these directions, Iteration 1 (what we have now) feels the most viable. I don't currently have any other ideas for how to design this, do you two? @croswell81 @damonmcc

@jackrosacker love having an example and those tables!
Iteration 1 is how we had it. While making the changes to have the flag-column-per-question structure of the final table, there's now a single id-column-per-question in that final table.
Iteration 3 is my favorite: one column of ID values per question. But I wonder if having column names like Hist Res - Adjacent would be better than Hist Res ID - 90', so that people can easily relate the column to the question and you won't have to maintain buffer values in alias strings.
If EARD review requires "exactly which dataset did that value come from", Iteration 2 seems like something we can add later in addition to Iteration 3.

The example is missing how many of these values would just be repeated, since any lot that has a historic resource will also be in the buffer, or any resource within 200 feet will also be within 90 feet.
My concern is this could potentially add dozens of fields since there are 10+ natural resource fields that go into NR shadow, and another 5-8 historic resources with two buffers, etc.
We should try to meet tomorrow when Jack is available.

@damonmcc - I had missed that we were dropping the per-dataset name/ID columns. I think what we had wanted was a flag per question, and a name/ID per dataset. That might be where some of the confusion re my extra tab in the excel sheet came from. In your current output, if values of more than one dataset are concatenated into a single name-per-question column, is there a way to tell what dataset each name is derived from?
From what I'm hearing, it sounds like what you've been working on currently is actually close-ish to iteration 3 in my comment above.
@croswell81 - that's my concern as well. Iteration 2 would mean a massive increase in the number of output columns.

@jackrosacker

From what I'm hearing, it sounds like what you've been working on currently is actually close-ish to iteration 3 in my comment above.

Yes but only because it's my first pass at prioritizing columns related to questions.

In your current output, if values of more than one dataset are concatenated into a single name-per-question column, is there a way to tell what dataset each name is derived from?

The only way to tell would be the custom prefix we add to each name (e.g. NYC: F. J. Berlenbach House). Using a prefix seems like the only way to avoid the explosion of columns in Iteration 2. Maybe that prefix should be a more explicit indication of the dataset it refers to?

Ideally, we would have the per question yes/no column (flag?), and a per dataset name/id column. This was the goal with the current request, but I think we must have crossed wires between the flag request with the desired name/id output. The would be consistent with what's currently in the beta release, with the fewest changes either for your team, or for PS to review.
I put time on our calendars for tomorrow @ 11 - might be best to screenshare?

@damonmcc forwarding the data requests noted during our last meeting with Planning Support. Adding notes re priority for each. We'd chatted about these as hypotheticals last week, and PS confirmed a number of them on Friday. Dumping here so you can add to existing or create new issues as needed.
Priority 1 (essential before go-live):

Develop data quality tests to check for things such as blank zoning category values, blank flag values in any field, etc.

Would be helpful to brainstorm together what other data checks could be run

Priority 2 (nice to have before go-live):

Filter CATS permits to only include PA/PB records

involves querying the "applicationid" column to read the first two characters of the alphanumeric id, and exclude anything that doesn't start with PA or PB
I've added this to Matt's Data Sources sheet as well
this was identified as an enhancement, but seems relatively straightforward to implement for go-live if possible

Priority 3 (could wait until after go-live if necessary):

for all E designation name/id fields - instead of returning just enumber, return a concatenated enumber - CEQR number (formatting and id order can be shifted, the important part was user feedback asking for the inclusion of the CEQR number as well)

Requested but on Hold

adding a value such as "n/a" when no value is returned in name/id field

you'd voiced some reasonable hesitations about this, and we're meeting with someone from BSA who might give us more clarity on this specific ask tomorrow. If this was confirmed, it would be relatively low priority

@damonmcc we just had a good tool critique meeting with BSA, and they raised one point that impacts data (sorry). When a lot is flagged for a dataset that has no name or id values, it's confusing for the user to get a "Yes" flag, but a blank field in the output CSV. The example dataset was the archaeological sensitivity areas, which have no identifiers in the source data.
It would be valuable to add a clause for the green_fast_track_bbl table that says: if the flag == "Yes" and name and id are both blank, populate the name/id field with a static value. Maybe the name of the dataset in a human-readable format?
This feels like a Priority 2 or 3 item as defined in the comment above.

@jackrosacker makes sense that we wouldn't want empty values next to a Yes flag. we actually do put Archaeological Areas in the ID column when a lot intersects with that. maybe the build they're seeing was from before we made that change
I just checked my latest build and selecting where archaeological_area_flag = 'Yes' and archaeological_area is null retuned no rows, so we should be good with that flag
I'll keep an eye out for any occurrences  of Yes w/ no ID

Ok, sounds great. The data they're seeing is from 4/25, so is definitely old

closing this since the major work of creating the data and app is complete!! 🎉🎉🎉
@fvankrieken started this issue for follow-up work to make GFT a more formal data product that we expect to build frequently


Problem installing package in an isolated environment

I'm trying to install a package in an AWS instance which is not accessible to the pypi.org.
I had to upload it to a private repository using s3pypi. (With all dependencies of course)
when trying to install said package, this is the stack trace I get:
Traceback (most recent call last):
  File "/opt/conda/lib/python3.7/site-packages/pip/_internal/cli/base_command.py", line 188, in main
    status = self.run(options, args)
  File "/opt/conda/lib/python3.7/site-packages/pip/_internal/commands/install.py", line 345, in run
    resolver.resolve(requirement_set)
  File "/opt/conda/lib/python3.7/site-packages/pip/_internal/legacy_resolve.py", line 196, in resolve
    self._resolve_one(requirement_set, req)
  File "/opt/conda/lib/python3.7/site-packages/pip/_internal/legacy_resolve.py", line 359, in _resolve_one
    abstract_dist = self._get_abstract_dist_for(req_to_install)
  File "/opt/conda/lib/python3.7/site-packages/pip/_internal/legacy_resolve.py", line 307, in _get_abstract_dist_for
    self.require_hashes
  File "/opt/conda/lib/python3.7/site-packages/pip/_internal/operations/prepare.py", line 215, in prepare_linked_requirement
    finder, self.build_isolation,
  File "/opt/conda/lib/python3.7/site-packages/pip/_internal/distributions/source.py", line 70, in prepare_distribution_metadata
    reqs = self.req.pep517_backend.get_requires_for_build_wheel()
  File "/opt/conda/lib/python3.7/site-packages/pip/_vendor/pep517/wrappers.py", line 71, in get_requires_for_build_wheel
    'config_settings': config_settings
  File "/opt/conda/lib/python3.7/site-packages/pip/_vendor/pep517/wrappers.py", line 162, in _call_hook
    raise BackendUnavailable
pip._vendor.pep517.wrappers.BackendUnavailable

Is there maybe a way to work around this?
Thanks

Hi! Thanks for filing this but this isn't the right place for this issue.
I think that python-list or python-tutor mailing lists, #python on Freenode (IRC) or StackOverflow would be better channels to get support for the issue you're facing.


BUG: Twilio Validation not correctly handling unauthenticated event - Delivery Status lambda

Description
We want to only accept Twilio delivery status callbacks from an authenticated Twilio request to keep our system secure. The validation for a bad payload is printing Authenticated Twilio request to the logs when a malformed and unauthenticated request is made to the lambda. The celery body is unable to be generated as expected and the message is pushed to the DLQ as expected, but the logging is incorrect (and probably the event handling when the event is malformed).

[ ] Ticket is understood, and QA has been contacted (if the ticket has a QA label).

Steps to Reproduce

Send the following test event to the Delivery Status lambda: {"kyle_message": "test"}
Observe an INFO log stating Authenticated Twilio request
Observe that the celery body is not generated and that a critical error is thrown for it
Observe that the message does go to the DLQ
Observe that there are no logs indicating that the validation failed

Workaround
None
Impact/Urgency

Low - this is an annoyance and poor logging but the code does the right thing with the request.
Expected Behavior

[ ] When I send a malformed and unauthenticated request through the delivery status processor lambda, then the request should return a 403 response, print an error log, and not accept the request.

QA Considerations

[ ] Exercise this lambda in both Perf/Staging to validate the expected behavior

Additional Info & Resources

NOTE This lambda expects a specific type of event to be passed into it, in this form (note all data including the base64 is fake). The event that was passed to the lambda was not in this format, so it may be that we need to fail out if a body does not come in this expected format. Certainly we should not be logging Authenticated Twilio request. Also the following payload correctly generated a 403 response.

Event: {'requestContext': {'elb': {'targetGroupArn': 'somearn'}}, 'httpMethod': 'POST', 'path': '/sms/deliverystatus', 'queryStringParameters': {}, 'headers': {'accept': '/', 'connection': 'close', 'content-length': '227', 'content-type': 'application/x-www-form-urlencoded; charset=utf-8', 'host': 'reverseproxydomain', 'i-twilio-idempotency-token': 'someguid', 'user-agent': 'TwilioProxy/1.1', 'x-amzn-trace-id': 'somevalue', 'x-forwarded-for': 'someips', 'x-forwarded-host': 'reverseproxyaddress', 'x-forwarded-port': '443', 'x-forwarded-proto': 'https', 'x-forwarded-scheme': 'https', 'x-home-region': 'us1', 'x-real-ip': 'someipaddress', 'x-twilio-signature': 'twiliosig'}, 'body': 'UmF3RmxvYXRJbmRlckRhdG09MjMwMzA5MjAyMSZTbXNTaWQ9U21zSUQmU21zU3RhdHVzPWFjY2VwdGVkJk1lc3NhZ2VTdGF0dXM9ZGVsaXZlcmVkJlRvPSUyQjExMTExMTExMTExJk1lc3NhZ2VTaWQ9U01kZWxpdmVyZWQmQWNjb3VudFNpZD1BQzExMQ=='}

Code block:

Note that the validate_twilio_event() method should be reviewed.
Logs:

Note that an error should have been thrown here (the following is from a PR showing the at-this-time existing code and code about to be merged, but both are error logs that should have been thrown).

This is the log sequence we see instead:

Hey team! Please add your planning poker estimate with Zenhub @cris-oddball @EvanParish @k-macmillan @kalbfled @ldraney @MackHalliday @mchlwellman

Still need to work on this.


Pull request test on feature branch

My first pull request on my own project.

test merging branch from PR


0.7.2

We managed to sneak a couple of regressions into 0.7.1, so I figured we should release 0.7.2 sooner rather than later.
(Generally, frequent small releases are self-evidently better, though that's tricky with some of the changes on the roadmap - server-side encapsulated CSS rendering, progressive enhancement, #1740 and so on. So if there are any obvious major bugs or regressions that need to be dealt with, we should try and identify them for a 0.7.3 release before 0.8.x work begins in earnest.)
Have I missed anything that needs to be in this release?

LGTM

:+1: I like frequent small releases especially after a major release.

This is now merged and published (http://cdn.ractivejs.org/0.7.2/ractive.js, http://cdn.ractivejs.org/latest/ractive.js etc, plus npm).

ractivejs.org needs desperately a blog post. The last one is from may/2014 (almost an year ago). People trying ractive can see that as a bad sign. 0.7 was a major release and i've seem not even a tweet about it :(

@guilhermeaiolfi you're totally right. I was trying to bring all the websites up-to-date before writing an announcement (a lot of stuff is/was as out of date as blog.ractivejs.org, e.g. the main website was running on Ractive 0.4.0...) but haven't managed to get that done yet. It is definitely on the todo list. Keep prodding us if it doesn't happen soon!


Coordinate publication subtypes with Semantic Science publication ontology?

From<EMAIL_ADDRESS>on July 29, 2009 15:44:09
We may want to pay attention to this: http://semanticscience.googlecode.com/svn/trunk/ontology/publication.owl
Original issue: http://code.google.com/p/information-artifact-ontology/issues/detail?id=37

From<EMAIL_ADDRESS>on August 11, 2009 10:29:03
The publication subtypes and document parts with similar names to the IAO ones are pretty compatible.  There
are some problems with their (textual) definitions, but asserting "same as" relations for them should be OK.  This
ontology has a lot of potentially useful relations that we might want to consider adding on our side (e.g.
hasAuthor, hasPublisher, hasDOI, etc.)

From<EMAIL_ADDRESS>on August 11, 2009 11:42:06
Agreed on the publications, but I disagree on the relations. While they may be useful
as shorthands, their definition can be meaningfully expressed through existing
relations. For example, author and publisher are roles that can be tied to a document
in a creation or publishing process, e.g.
document X is_specified_output_of some writing process and realizes some author role
borne by some 'Larry Hunter'

From michel.dumontier on August 11, 2009 11:43:23
Actually, it would be worth investigating how we can use more general relations in
place of the domain specific relations (hasAuthor, etc), and ensure that the
corresponding domain class is represented (e.g. Author / Author Role).
There is (probably) an existing mechanism to associate identifiers with documents.


parse error: Invalid escape at line 285, column 436

Hello,
I am attempting to use checklist_graph.sh on a Linux VM and connecting to my tenant/subs using Azure CLI.
When running the script the following error occurs:
parse error: Invalid escape at line 285, column 436

Hey @sijday thanks for logging the issue! For reference, line 285 is this:
success_result=$(az graph query -q "$graph_success_query" ${(z)mg_option} -o tsv 2>$error_file --query 'data[].id' | sort -u)

I haven't been to repro using bash or zsh. @sijday, which shell are you using? (you can verify with echo $SHELL).
Something interesting is that your error message says invalid escape, but I don't see any escape character in the previous line. @sijday, could you confirm that line 285 of your script looks exactly like the above?

Hello Jose,
Yes the line is the same in the script I have downloaded. I'm using bash.
I wondered whether the issue is with a character in one of the variable values.
Thanks,
Simon.

Here you go. I did also check separately that the vm could access the checklist url to check it wasn't an issue there.
./checklist_graph.sh --technology=lz --list-categories --debug DEBUG: Getting checklist from https://raw.githubusercontent.com/Azure/review-checklists/main/checklists/lz_checklist.en.json... parse error: Invalid escape at line 285, column 436 0: -

I had the same problem and I've manage to workaround it downloading the lz_checklist.en.json and modifying the regex on lines 285 and 286 from @'^(?:10|127|172\.(?:1[6-9]|2[0-9]|3[01])|192\.168)\..*' to @'^(?:10|127|172\.(?:1[6-9]|2[0-9]|3[01])|192\.168)\.'

Ah, that makes it much more clear. The problem is not in the script, but in the actual query documented in the LZ checklist. I have changed the escape characters in the LZ query so that they dont break when being processed. However, please be aware that the queries in the LZ checklist are not validated yet, this is an area where contributions would be highly appreciated.

@dgancedo sorry now I see your note. I used \\\\ in the checklist and that seemed to do the trick for me. Let me check by removing the asterisk...

@erjosito @dgancedo thanks I’ll give it another try

@sijday Ah, that makes it much more clear. The problem is not in the script, but in the actual query documented in the LZ checklist. I have changed the escape characters in the LZ query so that they dont break when being processed. However, please be aware that the queries in the LZ checklist are not validated yet, this is an area where contributions would be highly appreciated.

@erjosito how is it best to contribute graph queries as we refine them?

Any contribution would be highly appreciated :-)

Closing this now, the original problem seems to be fixed


Grant Yu-Chu write access

@lmz, @FlaviaTsang
Can you please give @ychtw write access to this repository? We found some bugs in some project cards.

Invite sent.

Looks like the invite was accepted, closing.


How do I run this?

Sorry, brand new to VSCode.  I just installed your extension but your animated gif doesn't show how you're actually opening it.

Launch from the menu bar View > Command Palette or ⇧⌘P (Ctrl+Shift+P on Windows and Linux).
Type Nuget: Open Gallery


Small about dialog revamp

Create About dialog contents using WPF,
Made the link clickable,
Slightly altered the text.

Thanks, I appreciate.


Set favicon as CMS global?

I'm vaguely aware that you don't want your favicon to be loaded in the same way as other images but it would be nice to be able to upload one without digging in the code.

Absolutely! The groundwork is in place because we're only using one asset for favicons, so it should be nice and simple for folks to change when it’s part of _data/site.json / wired up to the CMS.


How to add image on userchat

const clientMessage = props.actionProvider.createClientMesssage(image)
props.actionProvider.setClientMessage(clientMessage)
I want to change from message to image. Help me please

Hello. Please refrain from opening duplicate issues.


Update README.md

Minor fix regarding what this role does.

Thank you for your PR.

Glad you find this useful.
Cheers


feat(helm): add pod annotation with config checksum

PR Description
So the replicaSet (deployment, daemonset etc...) is automatically  restarted after config changes.
Which issue(s) this PR fixes

Notes to the Reviewer
PR Checklist

[ ] CHANGELOG.md updated
[ ] Documentation added
[ ] Tests updated
[ ] Config converters updated

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


Identity scaffolding should not require authentication for /Account/Logout

Logout.cshtml and Logout.cshtml.cs are used before and after the user is logged out. If user posts to the Logout page without a returnUrl value, the page displays a message: You have successfully logged out of the application. Unfortunately, the page is unreachable after logging out (i.e. requires authentication) due to the following line in the generated IdentityHostingStartup: https://github.com/aspnet/Scaffolding/blob/6558238ab8c2add9b75b8022d1445ed20c71c7a1/src/VS.Web.CG.Mvc/Templates/Identity/IdentityHostingStartup.cshtml#L80

Found while validating aspnet/Release#265

All related issues are closed. Closing this as well.


Prepare Release 1.1.0-m2

Purpose

$subject

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Comparison is base (74a2343) 26.52% compared to head (8e0fbe7) 80.18%.
Report is 1 commits behind head on main.

Additional details and impacted files
@@             Coverage Diff             @@
##             main    #1963       +/-   ##
===========================================
+ Coverage   26.52%   80.18%   +53.65%
===========================================
  Files         248        9      -239
  Lines       29054      550    -28504
  Branches     5150      246     -4904
===========================================
- Hits         7706      441     -7265
+ Misses      21100      109    -20991
+ Partials      248        0      -248

Flag
Coverage Δ

adapter
?

common-controller
?

config-deployer-service
?

idp-domain-service
80.18% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Correct offset type of GatherCmp intrinsic

As the other GatherCmp* variants and the actual implementation prototype  indicate , the offsets parameter must be int2, not int to account for the 2D access of a 2D texture.

@pow2clk : Thanks for your contribution! The author(s) have been notified to review your proposed change.


IndexOutOfBounds on viewing log of task that had error creating

Describe the bug
When a task has an error spinning up, you can still request the log from the front end.
Using the default Kubernetes Log retrieval option causes an IndexOutOfBounds if the task hasn't successfully been creating in Tekton
What project does this issue relate to?
Boomerang Flow
To Reproduce

Run a workflow with at least 1 task
Task cannot spin up due to various issues such as invalid rights
Attempt to view the tasks log in the activity window

Environment
Flow 3.4.1 on GCP 1.20
Additional Contenxt
java.lang.IndexOutOfBoundsException: Index 0 out of bounds for length 0
	at jdk.internal.util.Preconditions.outOfBounds(Preconditions.java:64) ~[?:?]
	at jdk.internal.util.Preconditions.outOfBoundsCheckIndex(Preconditions.java:70) ~[?:?]
	at jdk.internal.util.Preconditions.checkIndex(Preconditions.java:248) ~[?:?]
	at java.util.Objects.checkIndex(Objects.java:372) ~[?:?]
	at java.util.ArrayList.get(ArrayList.java:458) ~[?:?]
	at io.boomerang.kube.service.LogKubeServiceImpl.streamPodLog(LogKubeServiceImpl.java:63) ~[classes!/:2.4.3]
	at io.boomerang.service.LogServiceImpl.streamLogForTask(LogServiceImpl.java:60) ~[classes!/:2.4.3]
	at io.boomerang.controller.LogController.streamLogForTask(LogController.java:28) ~[classes!/:2.4.3]

Fixed in 6.4.5


Check for React presence when detecting WEBPACK_REACT type

To prevent incorrect detection when a vue project has webpack installed

Merge when ready


Add samanthajayasinghe as maintainer and approver

What type of PR is this?
(bug/feature/cleanup/documentation)
What this PR does / Why we need it?
Which Jira/Github issue(s) does this PR fix?
Resolves #
Special notes for your reviewer
Pre-checks (if applicable)

[ ] Ran unit tests locally
[ ] Validated the changes in a cluster
[ ] Included documentation changes with PR

Codecov Report

Merging #291 (be82f6d) into main (6411c2f) will not change coverage.
The diff coverage is n/a.

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #291   +/-   ##
=======================================
  Coverage   46.91%   46.91%
=======================================
  Files          53       53
  Lines        3568     3568
=======================================
  Hits         1674     1674
  Misses       1633     1633
  Partials      261      261

/lgtm