Add a complete lifecycle hook signal in LoginNodes user_data.sh and related support to avoid MITM waring

Description of changes

Add a complete lifecycle hooksignal in LoginNodes user_data.sh
Add a new lifecycle hook of launching
Add IAM policy for complete lifecycle hook signal
Modify related unit test to adopt new changes
Change Id of pool stack to make ASG name and lifecycle hook name unique

Checklist

[x] Make sure you are pointing to the right branch and add a label in the PR title (i.e. 2.x vs 3.x)
[x] Check all commits' messages are clear, describing what and why vs how.
[x] Make sure to have added unit tests or integration tests to cover the new/modified code.
[x] Check if documentation is impacted by this change.

Closed in favor of https://github.com/aws/aws-parallelcluster/pull/5520


W.I.P. -- Master Address Repository #49

Simple implementation of #149. It currently provides a script to go retrieve a csv for a given address. This is not finished, but P.R. submitted to receive feedback from the Tuesday meeting. Its an attempt at making a modular interface so other data sources can rapidly be added and data retrieved.
Example Usage:
bin/data.sh mar find_location --output ~/output.csv --params "location:Some street Address"

One other edit. Just to avoid confusion between our API and getting data from other api's, can you either move object based code into /ingestion/api or rename the /api folder to something like /api_requests or /api_data (both those names are clunky - I'm open to better suggestions). We will soon be adding code to serve our data through an API server that we set up so I want to keep the distinction clear.

I definitely hear you in regards to the opacity of the code -- to do it in a modular way and quickly I used a TON of reflection. I'll create a README to drop in the directory to explain the layout and how to "Create your own api class", as well as including an example on how to call the Api Connection functions directly, similary to what data.py does.
Just to clarify, my two end goals are to be able to get output csv's by using data.sh - - data.py should never be used directly - - and to have a Api Connection class for each api that other scripts can import if they want to have direct access to that api.
I'd like to move the /api directory to /sources , to imply that these are ways to connect to data sources. I want to keep it separate from the /ingestion module because my approach is significantly different.

/sources works for me!
I see how the shell file connects now - I'm not very proficient  with using these to run python myself. Mostly for clarity, I still recommend putting an example that has a hard-coded example address (you could use the 641 S St NW address of where we're meeting tonight) and hard-coded names of the api module and method. The data.sh can still be the generalizable version available in addition to a hard coded example .
Thanks!

Thanks for the additional feedback and testing - - that's very helpful. I'll work on remediating these issues tonight.

I've added a bunch of documentation and fixed most of the items you mentioned with the exception of dealing with not found addresses and get_condo_count. I would recommend that you go ahead and merge it and i'll submit another pull request next week with some updates.

@ehenry2172 I added a new issue: https://github.com/codefordc/housing-insights/issues/175. I added a little bit of additional documentation on how to run the data.py directly (since I'm on Windows, and a few others are as well). Be sure to pull the latest dev when you do so.
Can you add in a few unit tests as well when you're fixing those issues?
Thanks, this is great!


PHPLIB-599: Typing improvements

This PR handles PHPLIB-600 and PHPLIB-602, adding parameter types and return types where applicable.
Rules for phpcs have been updated to require argument types; no changes were made to return types as most of the classes are non-final (something we may want to address for 2.0).

Were most of these code changes manually done, or is this something phpcbf helped with after updating phpcs.xml.dist?

It was a mix of phpcbf and manual changes. Return types were added through manual labour as phpcs can't be configured to only enforce return types on private members or final classes.

@jmikola you are correct - it looks like tooling did indeed leave most (if not all) @return array and @param array type annotations in place. I've removed them manually where applicable.
As for @param mixed and @return mixed, these are currently required to let the tooling know that the parameter expects any type (as opposed to "we don't know what it accepts) and returns a mixed value (as opposed to "we don't know what it returns") respectively. As such, removing them will cause complaints from the tooling. On the other hand, leaving them in place now will also help us when we drop PHP < 8 and can add the mixed pseudo type as an actual type.
Last but not least, most of our classes are neither final nor @internal, so the number of return types we can add is severely limited. In addition to that, the concept of tentative return types added in PHP 8.1 is only available to built-in and extension classes, but not to user land code. This means that we don't have a nice mechanism of telling people that the return type of a method will change in a future version. Making these classes final is something we should re-evaluate for 2.0.


Sudden change in signature of Direct3D 11 functions with optional out-pointers

Which crate is this about?
windows
Crate version
0.46.0
Summary
The signature of a bunch of D3D11 functions changed suddenly. (In particular, the ones I have encountered are CreateRenderTargetView, CreatePixelShader, CreateInputLayout, CreateBuffer, CreateSamplerState, CreateBlendState). They all have in common that they have optional out-pointers which now have to be explicitly passed in. Is this intentional? These calls are now very verbose indeed to use, especially with all the Option-wrapping around the pointers.
Toolchain version/configuration
No response
Reproducible example
No response
Crate manifest
No response
Expected behavior
No response
Actual behavior
error[E0061]: this method takes 3 arguments but 2 arguments were supplied
     |
138  |         let vertex_shader = unsafe { device.CreateVertexShader(vertex_shader_bytes, None)? };
     |                                             ^^^^^^^^^^^^^^^^^^--------------------------- an argument of type `Option<*mut Option<ID3D11VertexShader>>` is missing

Additional comments
No response

I see. Nonetheless, the ergonomics of actually calling these functions now leaves something to be desired. Leaving aside the breakage, it's about the same as calling the raw function, except the out-pointers also have to be wrapped in Options. Between this and #2105 I now kinda wish windows-sys included vtables for these interfaces so I could wrap them myself. Optionality on the out-pointers doesn't even make sense for half of these functions. E.g. CreateSamplerState() doesn't say what it does or does not do when passed a null out-pointer.

While I sympathize that calling these functions isn't particularly convenient, this is something that some Direct3D developers indicated that they need. Direct3D is not primarily focused on productivity, but if you think some of these functions shouldn't be optional, you can certainly bring that up here.
cc @damyanp

Yep, feel free to suggest a more suitable approach to binding such parameters but I've mostly exhausted this space having spent a lot of time threading the needle between correctness and usability in this particular scenario.


Update go to 1.20

Magic mirror failed to merge https://github.com/open-cluster-management-io/multicloud-operators-subscription/pull/320 due to differing github actions.

/lgtm


docs: Clean up Diffbot docs

The Diffbot DocumentLoader page doesn't actually run for a number of reasons. This PR fixes it along with some light details on the Graph Transformer and Provider pages.
Full Changelog
Document Loader Page

Fixed the notebook so that it actually runs (missing required modules, env variables, etc..)
Added "open in colab" button like the Graph Transformer page

Graph Transformer Page

Fixed broken colab link
Moved "open in colab" button to below description so the description in the Graphs category page shows up correctly

Provider Page

Clarified explanations of Diffbot products
Added section and link to LangChain Graph Transformer page

@eyurtsev any chance you'd be able to review this PR? We've had a few folks reach out to us at Diffbot with issues following the langchain docs. Hoping to remedy that.

Messed up a rebase. Resetting...

Thanks @baskaryan! A little hard to tell from the notebook files what you changed. Anything I should know for future commits to make it easier to review?


Flutter pod install generates invalid path for .xcconfig files

i have issues with integrating flutter in my existing iOS(swift) application,
according to the docs in order to user flutter i added this in my pod file:
flutter_application_path = 'my_flutter_module/' eval(File.read(File.join(flutter_application_path, '.ios', 'Flutter', 'podhelper.rb')), binding)
i then run pod install and everything works fine, i manage to install my app and also to archive, but when my CI / CD server run the build it failed every time, when diving into the logs i identified that the flutter scripts generated code on all the .xcconfig files that looks like this
#include "/Users/my_name/my_project_name/my_flutter_module/.ios/Flutter/Generated.xcconfig"
this path is relative to my machine, but i expect that the path's will look like
#include "${PODS_xxx}/my_flutter_module/.ios/Flutter/Generated.xcconfig"
after some more digging i manage to find the script that generate this at:
/Users/my_name/flutter/packages/flutter_tools/templates/module/ios/library/Flutter.tmpl/podhelper.rb
CODE
post_install do |installer| installer.pods_project.targets.each do |target|     target.build_configurations.each do |config|         config.build_settings['ENABLE_BITCODE'] = 'NO'         next if  config.base_configuration_reference == nil         xcconfig_path = config.base_configuration_reference.real_path         File.open(xcconfig_path, 'a+') do |file|             file.puts "#include \"#{File.realpath(File.join(framework_dir, 'Generated.xcconfig'))}\""         end     end   end  end
i tried to modify it but there was no affect after pod install, any help will be appreciate.
note that i use the .gitignore from here
and all my development / other pods gets the wrong local path like this(my firebase pod):

I also encountered the same problem. Is there any solution?

I temporarily add Pods/Target Support Files/*.xcconfig to .gitignore to solve this problem.

what work for me in the end is to exclude all the pods from my git and run "pod install" before every build on the server.

This should be resolved now as of https://github.com/flutter/flutter/pull/36793.  On the master branch, try updating your project.  See the instructions at https://github.com/flutter/flutter/wiki/Add-Flutter-to-existing-apps#add-your-flutter-app-to-your-podfile and https://github.com/flutter/flutter/wiki/Upgrading-Flutter-added-to-existing-iOS-Xcode-project.
If you are still seeing problems, please re-open this or file a new issue!


Search Student by Name

The website should ideally have the feature to search students by name. This has come in handy when students want to know if their assignments have gone through and often ask TA's to check

Not present priority


Continue - app_projects_mini_v0.0.4 component

#4
todo

[x] Update app_projects_mini_v0.0.4

@input 📦 app_projects_mini_v0.0.4 from #12
[x] Randomize project on home_page
@output :package: app_projects_mini_v0.0.5 from comment

tasks - 2024.01.14

Update app_projects_mini_v0.0.4

Randomize project on home_page - 12min
Recorded worklog video - 13min
Created and logged task - 5min
@output 📦 app_projects_mini_v0.0.5

worklog
worklog-192

feedback

proposal


Stop or pause everything in background when flag popup is opened

Do not allow scrolling

Maybe user the Anchor component in Ant Design

either stop the user from scrolling or have the flag popup move in unison with the message that is getting flagged


[Bug]

The problem
on new beta core integration stop working
logs clear

Your device model

Firmware type
Stock
Firmware version

Integration version with the issue

Method of the integration installation
HACS
What version and type of Home Assistant installation do you use
2022.12.0b1
Is there anything useful in the logs?
No response
Diagnostics information
No response
Additional information
No response

I have the same issue, it looks like HA core went to different version of asusrouter, below is the error message that I have.
Logger: homeassistant.util.package
Source: util/package.py:98
First occurred: 7:00:08 PM (3 occurrences)
Last logged: 7:00:15 PM
Unable to install package asusrouter==0.12.0: ERROR: Cannot install asusrouter==0.12.0 because these package versions have conflicting dependencies. ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts [notice] A new release of pip available: 22.3 -> 22.3.1 [notice] To update, run: pip install --upgrade pip

@milandzuris,
@thewazz1,
Thanks for your reports! This issue will be fixed as soon as possible.

@milandzuris,
@thewazz1,
Thanks for your reports! This issue will be fixed as soon as possible.

Please add this fix to beta

Version 0.13.1 of the integration is already available in HACS. Please, check it out - it should work with HA 2022-12

it works


describe burn minimum unit

https://github.com/gridcoin/Gridcoin-Research/blame/67ed882b100aa315ab0ab27e13d95623295890df/src/main.cpp#L8482

Kind of a vague question would help if your pointing more towards the answer you would like.
I know we have a minimum TX fee of 0.00010000 (10000 satoshis)
When basic contracts are added the burn fee is 0.00010000;
When this of course gets called it is passed to a rounding function and ends up as 0.00011000.
When a project is added the burn fee is 0.10000000.
Im assuming (could be wrong) that any burn fee would be subject to the minimum TX fee as a minimum.

The burn amount must only be greater than zero. It can be as small as 1 Sat. However the transaction fee must be paid based on the size of the transaction.

thanks for clearing that up brod makes sense.

The 0.1 burn for project beacon is purely arbitrary. It is hard-coded into the sending procedure and never checked again.

The burn amount must only be greater than zero. It can be as small as 1 Sat.

WriteCache(sMessageType,sMessageKey+";BurnAmount",RoundToString(dAmount,2),nTime);
can be changed to
WriteCache(sMessageType,sMessageKey+";BurnAmount",ToString(dAmount),nTime);
Is it right? Then what needs a rounding function?

Make a PR, @WhiteCat6142 . However small and insignificant it is, you will become a contributor if it passes.

@denravonska can we close this?

Even thou it is not really solved, I do not see a point of this issue.

Burn amounts are no longer stored.


Types

The problem
On the type system page there are TODOs to be replaced by text explaining Modules, Packages, and Grammars. #2232
Solution provided
Have put in a short explanation and a link to the page specifically about that subject.

Thank you!


Change MockResponse body to Source.

Sigh. More hanging tests.

LGTM

I went full-buffer which means you can stream anymore. Do we care?
The problem is that some tests re-enqueue the same MockResponse instance for multiple requests. This worked if and only if you were using a Buffer/byte[] but not if you were specifying an InputStream (which nobody was internally). Do we need to support one-shot streaming responses?

Let's kill one-shot streaming responses. They were there for testing large payloads.


feat: 018

Add feat: 018

Thank you for this PR! I’m on vacation so I’ll take a look later today ASAP. Another contributor has changed the way config.json is structured, so if you have time, please rebase against the latest master and push again! ❤

Nice animation you did there. I've resolved the conflicts locally and added a few extra instructions on README.md on how to keep your fork up to date with the upstream master. This PR is merged! 👍


promote exit channel handle reference to top level

Modified to wrap up the exit handle reference collection into a struct and push into the top level provider in order to provide continutity between multiple calls to the NewVM method.


Fix WS chat for invalid member doesn't show error

Explanation of Change

When inviting an invalid member to WS, we want to show a create chat error.
Fixed Issues

$ https://github.com/Expensify/App/issues/51316
PROPOSAL: https://github.com/Expensify/App/issues/51316#issuecomment-2431657472
Tests

Same as QA Steps

[x] Verify that no errors appear in the JS console

Offline tests

Same as QA Steps
QA Steps

Open a workspace members page
Invite an invalid user (e.g., +1<PHONE_NUMBER>)
Wait until it fails
Go back to LHN
Open the WS chat of the invalid user
Verify it shows error and you can clear/close it

[x] Verify that no errors appear in the JS console

PR Author Checklist

[x] I linked the correct issue in the ### Fixed Issues section above
[x] I wrote clear testing steps that cover the changes made in this PR

[x] I added steps for local testing in the Tests section
[x] I added steps for the expected offline behavior in the Offline steps section
[x] I added steps for Staging and/or Production testing in the QA steps section
[x] I added steps to cover failure scenarios (i.e. verify an input displays the correct error message if the entered data is not correct)
[x] I turned off my network connection and tested it while offline to ensure it matches the expected behavior (i.e. verify the default avatar icon is displayed if app is offline)
[x] I tested this PR with a High Traffic account against the staging or production API to ensure there are no regressions (e.g. long loading states that impact usability).

[x] I included screenshots or videos for tests on all platforms
[x] I ran the tests on all platforms & verified they passed on:

[x] Android: Native
[x] Android: mWeb Chrome
[x] iOS: Native
[x] iOS: mWeb Safari
[x] MacOS: Chrome / Safari
[x] MacOS: Desktop

[x] I verified there are no console errors (if there's a console error not related to the PR, report it or open an issue for it to be fixed)
[x] I followed proper code patterns (see Reviewing the code)

[x] I verified that any callback methods that were added or modified are named for what the method does and never what callback they handle (i.e. toggleReport and not onIconClick)
[x] I verified that comments were added to code that is not self explanatory
[x] I verified that any new or modified comments were clear, correct English, and explained "why" the code was doing something instead of only explaining "what" the code was doing.
[x] I verified any copy / text shown in the product is localized by adding it to src/languages/* files and using the translation method

[x] If any non-english text was added/modified, I verified the translation was requested/reviewed in #expensify-open-source and it was approved by an internal Expensify engineer. Link to Slack message:

[x] I verified all numbers, amounts, dates and phone numbers shown in the product are using the localization methods
[x] I verified any copy / text that was added to the app is grammatically correct in English. It adheres to proper capitalization guidelines (note: only the first word of header/labels should be capitalized), and is either coming verbatim from figma or has been approved by marketing (in order to get marketing approval, ask the Bug Zero team member to add the Waiting for copy label to the issue)
[x] I verified proper file naming conventions were followed for any new files or renamed files. All non-platform specific files are named after what they export and are not named "index.js". All platform-specific files are named for the platform the code supports as outlined in the README.
[x] I verified the JSDocs style guidelines (in STYLE.md) were followed

[x] If a new code pattern is added I verified it was agreed to be used by multiple Expensify engineers
[x] I followed the guidelines as stated in the Review Guidelines
[x] I tested other components that can be impacted by my changes (i.e. if the PR modifies a shared library or component like Avatar, I verified the components using Avatar are working as expected)
[x] I verified all code is DRY (the PR doesn't include any logic written more than once, with the exception of tests)
[x] I verified any variables that can be defined as constants (ie. in CONST.js or at the top of the file that uses the constant) are defined as such
[x] I verified that if a function's arguments changed that all usages have also been updated correctly
[x] If any new file was added I verified that:

[x] The file has a description of what it does and/or why is needed at the top of the file if the code is not self explanatory

[x] If a new CSS style is added I verified that:

[x] A similar style doesn't already exist
[x] The style can't be created with an existing StyleUtils function (i.e. StyleUtils.getBackgroundAndBorderStyle(theme.componentBG))

[x] If the PR modifies code that runs when editing or sending messages, I tested and verified there is no unexpected behavior for all supported markdown - URLs, single line code, code blocks, quotes, headings, bold, strikethrough, and italic.
[x] If the PR modifies a generic component, I tested and verified that those changes do not break usages of that component in the rest of the App (i.e. if a shared library or component like Avatar is modified, I verified that Avatar is working as expected in all cases)
[x] If the PR modifies a component related to any of the existing Storybook stories, I tested and verified all stories for that component are still working as expected.
[x] If the PR modifies a component or page that can be accessed by a direct deeplink, I verified that the code functions as expected when the deeplink is used - from a logged in and logged out account.
[x] If the PR modifies the UI (e.g. new buttons, new UI components, changing the padding/spacing/sizing, moving components, etc) or modifies the form input styles:

[x] I verified that all the inputs inside a form are aligned with each other.
[x] I added Design label and/or tagged @Expensify/design so the design team can review the changes.

[x] If a new page is added, I verified it's using the ScrollView component to make it scrollable when more elements are added to the page.
[x] If the main branch was merged into this PR after a review, I tested again and verified the outcome was still expected according to the Test steps.

Screenshots/Videos

Android: Native

https://github.com/user-attachments/assets/293e15ec-4108-42a4-a464-82775dd239dd

Android: mWeb Chrome

https://github.com/user-attachments/assets/818ead0a-1172-4dc3-a41c-7c4865ccc7ee

iOS: Native

https://github.com/user-attachments/assets/34d37320-e49e-4a97-8a34-8f690627f4b2

iOS: mWeb Safari

https://github.com/user-attachments/assets/487a31f3-f82f-4a9e-9a46-7d3f27256722

MacOS: Chrome / Safari

https://github.com/user-attachments/assets/5a958548-3aa1-4d77-9526-7d5bad7cd58a

MacOS: Desktop

https://github.com/user-attachments/assets/7f8e8f6e-890b-4204-99c2-6630d26315d8

Reviewer Checklist

[x] I have verified the author checklist is complete (all boxes are checked off).
[x] I verified the correct issue is linked in the ### Fixed Issues section above
[x] I verified testing steps are clear and they cover the changes made in this PR

[x] I verified the steps for local testing are in the Tests section
[x] I verified the steps for Staging and/or Production testing are in the QA steps section
[x] I verified the steps cover any possible failure scenarios (i.e. verify an input displays the correct error message if the entered data is not correct)
[x] I turned off my network connection and tested it while offline to ensure it matches the expected behavior (i.e. verify the default avatar icon is displayed if app is offline)

[x] I checked that screenshots or videos are included for tests on all platforms
[x] I included screenshots or videos for tests on all platforms
[x] I verified tests pass on all platforms & I tested again on:

[x] Android: Native
[x] Android: mWeb Chrome
[x] iOS: Native
[x] iOS: mWeb Safari
[x] MacOS: Chrome / Safari
[x] MacOS: Desktop

[x] If there are any errors in the console that are unrelated to this PR, I either fixed them (preferred) or linked to where I reported them in Slack
[x] I verified proper code patterns were followed (see Reviewing the code)

[x] I verified that any callback methods that were added or modified are named for what the method does and never what callback they handle (i.e. toggleReport and not onIconClick).
[x] I verified that comments were added to code that is not self explanatory
[x] I verified that any new or modified comments were clear, correct English, and explained "why" the code was doing something instead of only explaining "what" the code was doing.
[x] I verified any copy / text shown in the product is localized by adding it to src/languages/* files and using the translation method
[x] I verified all numbers, amounts, dates and phone numbers shown in the product are using the localization methods
[x] I verified any copy / text that was added to the app is grammatically correct in English. It adheres to proper capitalization guidelines (note: only the first word of header/labels should be capitalized), and is either coming verbatim from figma or has been approved by marketing (in order to get marketing approval, ask the Bug Zero team member to add the Waiting for copy label to the issue)
[x] I verified proper file naming conventions were followed for any new files or renamed files. All non-platform specific files are named after what they export and are not named "index.js". All platform-specific files are named for the platform the code supports as outlined in the README.
[x] I verified the JSDocs style guidelines (in STYLE.md) were followed

[x] If a new code pattern is added I verified it was agreed to be used by multiple Expensify engineers
[x] I verified that this PR follows the guidelines as stated in the Review Guidelines
[x] I verified other components that can be impacted by these changes have been tested, and I retested again (i.e. if the PR modifies a shared library or component like Avatar, I verified the components using Avatar have been tested & I retested again)
[x] I verified all code is DRY (the PR doesn't include any logic written more than once, with the exception of tests)
[x] I verified any variables that can be defined as constants (ie. in CONST.js or at the top of the file that uses the constant) are defined as such
[x] If a new component is created I verified that:

[x] A similar component doesn't exist in the codebase
[x] All props are defined accurately and each prop has a /** comment above it */
[x] The file is named correctly
[x] The component has a clear name that is non-ambiguous and the purpose of the component can be inferred from the name alone
[x] The only data being stored in the state is data necessary for rendering and nothing else
[x] For Class Components, any internal methods passed to components event handlers are bound to this properly so there are no scoping issues (i.e. for onClick={this.submit} the method this.submit should be bound to this in the constructor)
[x] Any internal methods bound to this are necessary to be bound (i.e. avoid this.submit = this.submit.bind(this); if this.submit is never passed to a component event handler like onClick)
[x] All JSX used for rendering exists in the render method
[x] The component has the minimum amount of code necessary for its purpose, and it is broken down into smaller components in order to separate concerns and functions

[x] If any new file was added I verified that:

[x] The file has a description of what it does and/or why is needed at the top of the file if the code is not self explanatory

[x] If a new CSS style is added I verified that:

[x] A similar style doesn't already exist
[x] The style can't be created with an existing StyleUtils function (i.e. StyleUtils.getBackgroundAndBorderStyle(theme.componentBG)

[x] If the PR modifies code that runs when editing or sending messages, I tested and verified there is no unexpected behavior for all supported markdown - URLs, single line code, code blocks, quotes, headings, bold, strikethrough, and italic.
[x] If the PR modifies a generic component, I tested and verified that those changes do not break usages of that component in the rest of the App (i.e. if a shared library or component like Avatar is modified, I verified that Avatar is working as expected in all cases)
[x] If the PR modifies a component related to any of the existing Storybook stories, I tested and verified all stories for that component are still working as expected.
[x] If the PR modifies a component or page that can be accessed by a direct deeplink, I verified that the code functions as expected when the deeplink is used - from a logged in and logged out account.
[x] If the PR modifies the UI (e.g. new buttons, new UI components, changing the padding/spacing/sizing, moving components, etc) or modifies the form input styles:

[x] I verified that all the inputs inside a form are aligned with each other.
[x] I added Design label and/or tagged @Expensify/design so the design team can review the changes.

[x] If a new page is added, I verified it's using the ScrollView component to make it scrollable when more elements are added to the page.
[x] If the main branch was merged into this PR after a review, I tested again and verified the outcome was still expected according to the Test steps.
[x] I have checked off every checkbox in the PR reviewer checklist, including those that don't apply to this PR.

Screenshots/Videos

Android: Native

Android: mWeb Chrome

iOS: Native

iOS: mWeb Safari

MacOS: Chrome / Safari
https://github.com/user-attachments/assets/30a2dadd-6053-4d32-88f9-9445994b10b3

MacOS: Desktop

:hand: This PR was not deployed to staging yet because QA is ongoing. It will be automatically deployed to staging after the next production release.


How to modify BashOperator task's environment dinamically

Hello guys!
Here we use a bunch of python scripts to do our ETLs. Scripts that takes parameters like date_ini and date_end from the environment.
In Airflow we have the execution_date, that represents the date which the task instance is running for (our date_ini).
The execution_date is only available on the task's context, which makes sense. But, some way I need to get that value, make some magic and put a date_end in the environment, so I can continue with the os.getenv() and don't need to modify my python scripts.
Do you guys have some tip to achieve this?
I already tried to set the pre_execute in my task passing a method which modifies the context, but no success :disappointed:.
def set_env_var(context):
    return context.update({"my_env_var": "MY_env_var_value"})

bash_command_str = "python -c \"import os; print os.getenv('my_env_var');\""
bashTask = BashOperator(dag = dag, task_id = "asas", bash_command = bash_command_str)
bashTask.pre_execute = set_env_var

I don't know exactly if is it, but I just guess whether the sort of these lines were changed... Maybe the pre_execute would work.
https://github.com/airbnb/airflow/blob/master/airflow/models.py#L983-L984
Thanks in advance!

Check out the env parameter in BashOperator
http://pythonhosted.org/airflow/_modules/bash_operator.html#BashOperator
I just added a PR (https://github.com/airbnb/airflow/pull/776/files) to make this parameter templated, meaning the values of the dict can receive jinja as in env = {'execution_date': "{{ execution_date.isoformat() }}"} and will get replaced at runtime.
Here's the list of what you can use:
http://pythonhosted.org/airflow/code.html#macros

I tried the env parameter, but today I can't update the values at runtime.
If I understood, after the PR be merged I will be able to do:
from airflow.operators import BashOperator

customEnv = {
'date_ini': "{{ execution_date.isoformat() }}",
'date_end': "{{ execution_date + dag.schedule_interval }}"
}
env = dict(os.environ, **customEnv)
task = BashOperator(
            ...
            env=env
        )

# on the etl scripts
import os
...
date_ini = os.getenv('date_ini')
date_end = os.getenv('date_end')
...

Exactly, I just merged the PR


Differentiate close and update signature checks

with different MessageTypeId's.  This closes #1149.

$ pipdeptree --warn fail

is now complaining.


Azure keyvault sign() function gives errors on size of bytes for value

Im using the official SDK docs which states only that the 'value' variable passed to the sign function should be in bytes.  I attempted to use the sign() function but got an error saying Invalid length of 'value': 44 bytes. ES256 requires 32 bytes, encoded with base64url.  so then I take my 32bytes and pass those 32bytes to the urlsafe_b64encode and use that result in value for the sign() function, and I still get same error.  I suspect that we are actually not supposed to base64urlencode it first?  Please confirm if we are supposed to take the direct 32byte hash and supply that to the sign() function?  Or if indeed some base64 needs to happen in some manner?
This can easily be duplicated
#!/usr/bin/env python3

from azure.keyvault import KeyVaultClient
from msrestazure.azure_active_directory import MSIAuthentication
from base64 import urlsafe_b64encode
from hashlib import blake2b
from binascii import unhexlify

kvclient = KeyVaultClient(MSIAuthentication(resource='https://vault.azure.net'));
op = urlsafe_b64encode(blake2b(unhexlify('00112233'), digest_size=32).digest())
print(op)
sig = kvclient.sign('https://REDACTED1.vault.azure.net', 'REDACTED2', 'REDACTED3', 'ES256', op)

$ ./aclient.py
b'QO46UZnfhhh93VCTkagtPzWj1Z-4PvmzmKdsmA4fepY='
Traceback (most recent call last):
  File "./aclient.py", line 12, in <module>
    sig = kvclient.sign('https://REDACTED1.vault.azure.net', 'REDACTED2', 'REDACTED3', 'ES256', op)
  File "/home/dev/hsm/lib/python3.6/site-packages/azure/keyvault/v7_0/key_vault_client.py", line 1022, in sign
    raise models.KeyVaultErrorException(self._deserialize, response)
azure.keyvault.v7_0.models.keyvault.v7_0.models.key_vault_error_py3.KeyVaultErrorException: (BadParameter) Invalid length of 'value': 44 bytes. ES256 requires 32 bytes, encoded with base64url.

on second thought - that error message is probably only there to support the REST API, which does require a text string of base64 encoded version of 32bytes

@schaabs can you take a look into this issue please? Thanks!

@tezzigator I believe you're correct that the error message is misleading here.  The SDK does indeed just take in bytes and it handles the encoding for you.  I'm not sure but I think the issue here is the hash algorithm you're using.  I believe that the ES256 sign algorithm expects a digest created with SHA256.  Could you try changing your hash algorithm to SHA256 instead of blake2 and see if that fixes your issue?

Hi - i actually have it working after i figured out the message was a red herring for my particular instance (using SDK and not REST-API).  my apoligies if my 2nd message wasnt clear as it was meant to mean that I had it all figured out and was working as that message only applies to REST


⚠ Bofrost.at has degraded performance

In 637380b, Bofrost.at (https://www.bofrost.at) experienced degraded performance:

HTTP code: 200
Response time: 2429 ms

Resolved: Bofrost.at performance has improved in 3eee76e after 20 minutes.


Add retry logic to Az commands

Task name: VstsAzureHelpers
Description: Adds Retry logic via Retry-Command function (retry 5 times with a 5 second delay in between each retry) when running Azure commands: Connect-AzAccount and Set-AzContext. This is added to take into account ADO token replication delay, which is currently causing invalid access token errors when using Azure tasks such as Azure PowerShell.

@microsoft-github-policy-service agree [company=Microsoft]

@microsoft-github-policy-service agree company="Microsoft"

@PavloAndriiesh how do trigger the CI builds / merge my changes?

/azp run

@PavloAndriiesh it looks like some of the builds are failing because the task versions weren't updated (tasks weren't changed but the common library they use was modified).


Publish build cache

https://docs.gradle.org/current/userguide/build_cache.html
https://docs.gradle.com/enterprise/tutorials/caching/

Just pick up a6e39a4a46155e0676f06477dc0171ff7c8f406c to main.


Backport "Don't give a hard error when the end-user specifies RUSTC_BOOTSTRAP=crate_name"

Backports https://github.com/rust-lang/cargo/pull/9365, fixing https://github.com/rust-lang/cargo/issues/9362.

r? @alexcrichton
(rust-highfive has picked a reviewer for you, use r? to override)

:warning: Warning :warning:

Pull requests are usually filed against the master branch for this repo, but this one is against rust-1.52.0. Please double check that you specified the right target!

r? @ehuss

Thanks!
@bors r+

:pushpin: Commit 2dbeda8b82576bbce4cdefe6fbe00ae4d8f7096c has been approved by ehuss

:hourglass: Testing commit 2dbeda8b82576bbce4cdefe6fbe00ae4d8f7096c with merge 69767412acbf7f64773427b1fb53e45296712c3c...

:sunny: Test successful - checks-actions
Approved by: ehuss
Pushing 69767412acbf7f64773427b1fb53e45296712c3c to rust-1.52.0...


Verification functions do not check the consistency between the NID of the supplied leaves and the queried NID

Problem
None of the current proof verification functions i.e., VerifyNamespace and VerifyInclusion verify the consistency between the namespace ID of the leaves and the queried name ID (this inconsistency gets caught later in the root calculation). This can be addressed by incorporating a check at the beginning of the functions, which not only makes the code more clear and easier to maintain and debug, but also saves a significant amount of time by preventing unnecessary root calculations from invalid inputs and allowing for early returns from the function call.

Update:
VerifyNamespace does check the NID of the leaves against the queried NID in this line prior to hashing. . Nothing to be done in this method.
VerifyInclusion does not have to do any check since leaves argument are not namespaced.
The only method that can do the check but does not is the newly exposed VerifyLeafHashes. However, incorporating this check depends on our decision on the following issue https://github.com/celestiaorg/nmt/issues/110, so going to mark it as blocked until the other one gets concluded.

Update: VerifyLeafHashes should be revised to include a check that ensures the namespace of the leafHashes parameter (if it is present) matches the queried namespace. This required behavior should remain consistent regardless of whether the proof is a partial absence proof or a full absence proof (in the case of an absence proof, leafHashes will be empty). I will proceed with implementing this modification.


Migrations keep looping / timeout on permissions scripts on v1.7.0

Describe the bug
Grate successfully connects to the DB, checks what scripts to run, successfully runs procedures and indices scripts but then timeouts when running permissions scripts. This started happening when we upgraded from 1.6.2 to 1.7.0. Downgrading back to 1.6.2 solved the issue. Target database is an Azure SQL Database and authentication is via access token.
To Reproduce
I will continue to investigate to come up with exact steps but so far what I know is that it timeouts on permissions scripts after upgrading to 1.7.0.
Expected behavior
Permission scripts are run successfully since they were run successfully in 1.6.2.
Desktop (please complete the following information):

OS: Windows Server 2019
Version: 1.7.0

Interestingly, the first time I ran grate, it seemed to run some internal migrations also:
> dotnet grate --connectionstring "Server=***.database.windows.net;Database=kimtest" --accesstoken $sqltoken
Initializing connections.
Running grate v1.7.0 (build date 04/29/2024 17:52:09) against ***.database.windows.net - kimtest.
Looking in . for scripts to run.
Please press enter when ready to kick...

================================================================================
Setup, Backup, Create/Restore/Drop
================================================================================
================================================================================
Grate Structure
================================================================================
Initializing connections.
Running grate v1.7.0 (build date 04/29/2024 17:52:09) against ***.database.windows.net - kimtest.
Looking in C:\Users\***\AppData\Local\Temp\tmpeicmwq.tmp for scripts to run.
================================================================================
Setup, Backup, Create/Restore/Drop
================================================================================
================================================================================
Grate Structure
================================================================================
================================================================================
Versioning
================================================================================
 Migrating kimtest from version <IP_ADDRESS> to 1.7.0.
 Versioning kimtest database with version 1.7.0.
================================================================================
Migration Scripts
================================================================================
Skipping 'BeforeMigration', beforeMigration does not exist.
Skipping 'AlterDatabase', alterDatabase does not exist.
Skipping 'Run After Create Database', runAfterCreateDatabase does not exist.
Skipping 'Run Before Update', runBeforeUp does not exist.

Looking for Update scripts in "C:\Users\***\AppData\Local\Temp\tmpeicmwq.tmp\up". These should be one time only scripts.
--------------------------------------------------------------------------------
  Running 'grate-internal/01_create_schema_grate.ENV.GrateInternalBoostrap-a61456d0-e00a-4933-b692-c6a5d7d51539.sql'.
  Running 'grate-internal/02_create_scripts_run_table.sql'.
  Running 'grate-internal/03_create_scripts_run_errors_table.sql'.
  Running 'grate-internal/04_create_version_table.sql'.
--------------------------------------------------------------------------------

Skipping 'Run First After Update', runFirstAfterUp does not exist.
Skipping 'Functions', functions does not exist.
Skipping 'Views', views does not exist.
Skipping 'Stored Procedures', sprocs does not exist.
Skipping 'Triggers', triggers does not exist.
Skipping 'Indexes', indexes does not exist.
Skipping 'Run after Other Anytime Scripts', runAfterOtherAnyTimeScripts does not exist.
Skipping 'Permissions', permissions does not exist.
Skipping 'AfterMigration', afterMigration does not exist.

grate v1.7.0 (build date 04/29/2024 17:52:09) has grated your database (kimtest)! You are now at version 1.7.0. All changes and backups can be found at "C:\Users\******\gratetest\scripts\output\migrations\kimtest\***.database.windows.net\2024-04-30T12_26_42.3776439_03_00".

Initializing connections.
Running grate v1.7.0 (build date 04/29/2024 17:52:09) against ***.database.windows.net - kimtest.
Looking in C:\Users\***\AppData\Local\Temp\tmpdh34or.tmp for scripts to run.
================================================================================
Setup, Backup, Create/Restore/Drop
================================================================================
================================================================================
Grate Structure
================================================================================
================================================================================
Versioning
================================================================================
 Migrating kimtest from version <IP_ADDRESS> to 1.7.0.
 Versioning kimtest database with version 1.7.0.
================================================================================
Migration Scripts
================================================================================
Skipping 'BeforeMigration', beforeMigration does not exist.
Skipping 'AlterDatabase', alterDatabase does not exist.
Skipping 'Run After Create Database', runAfterCreateDatabase does not exist.
Skipping 'Run Before Update', runBeforeUp does not exist.

Looking for Update scripts in "C:\Users\***\AppData\Local\Temp\tmpdh34or.tmp\up". These should be one time only scripts.
--------------------------------------------------------------------------------
  Running '02_create_scripts_run_table.sql'.
  Running '03_create_scripts_run_errors_table.sql'.
  Running '04_create_version_table.sql'.
--------------------------------------------------------------------------------

Skipping 'Run First After Update', runFirstAfterUp does not exist.
Skipping 'Functions', functions does not exist.
Skipping 'Views', views does not exist.
Skipping 'Stored Procedures', sprocs does not exist.
Skipping 'Triggers', triggers does not exist.
Skipping 'Indexes', indexes does not exist.
Skipping 'Run after Other Anytime Scripts', runAfterOtherAnyTimeScripts does not exist.
Skipping 'Permissions', permissions does not exist.
Skipping 'AfterMigration', afterMigration does not exist.

grate v1.7.0 (build date 04/29/2024 17:52:09) has grated your database (kimtest)! You are now at version 1.7.0. All changes and backups can be found at "C:\Users\******\gratetest\scripts\output\migrations\kimtest\***.database.windows.net\2024-04-30T12_26_44.3836389_03_00".

Initializing connections.

...

Hi, @kimjamia .
I understand this can be a bit confusing. I intended to hide the logs for the internal migrations/bootstrapping. As part of #482 , I've started using grate to handle the grate internal migration tables as well, to be able to evolve this structure as well. And, what tool best to handle table migrations than grate?
It's expected that there will be two internal migrations the first time you run grate 1.7.0+. This creates:

A set of "grate internal" tables that handle the versioning of the grate* tables
The grate tables, if they do not exist, using a migration that stores the history in the tables created above.

These will indeed be run from two different folders in the tmp directory.
After the two internal migrations has run, does not the grate migration (of your own database scripts) run as normal?

For what it's worth, I'm seeing the same behavior with permissions scripts on 1.7.0 against Azure SQL Managed Instance using normal SQL Authentication. Downgrading to 1.6.2 resolves the issue.

Thanks for chiming in, @DonalLOConnor !
To help me understand better, could you please give me some details about how you run grate? Is it the normal command-line version? Or docker? Are you also on Windows?
Could you please provide me with the command-line you use, and, also the connection string (without the secrets of course). I'm very interested in which type of authentication mechanism you are using. Is it Azure AD (aka Microsoft Entra) auth, or SQL server username and password (ref e.g. https://learn.microsoft.com/en-us/sql/connect/ado-net/sql/azure-active-directory-authentication?view=sql-server-ver16#setting-microsoft-entra-authentication)
Additionally, after you see the output from the two "internal grate migrations", which are expected, does your permission scripts not run too? Does it hang? Timeout? What happens? Could you please provide me with a full output of the run (you can of course anonymise it if you have anything you don't want to share with the whole world)? It will make it easier to understand what happens, and solve the problem.
Is the problem there in grate 1.7.2 also (released yesterday)?

Erik,
I’m running grate as part of an Azure DevOps pipeline on a Microsoft-hosted build agent running on Microsoft’s latest Ubuntu image, with the following steps.
Install grate: dotnet tool install –global grate
Ensure the shell script checked out of git is runnable: chmod 775 ./dev.noninteractive.sh
Run the shell script containing the grate command: ./dev.noninteractive.sh
The contents of the shell script, dev.noninteractive.sh, are below.
#!/bin/bash
if [[ -n "$GR_DEV_IDSERVICETOCDBDATAPIPELINE_USERNAME" && -n "$GR_DEV_IDSERVICETOCDBDATAPIPELINE_PASSWORD" && -n "$GR_DEV_DB_ADMIN_USERNAME" && -n "$GR_DEV_DB_ADMIN_PASSWORD" && -n "$GR_DEV_DB_DBNAME" && -n "$GR_DEV_DB_PUBLIC_ENDPOINT" ]]; then
echo "All needed environment variables are defined. Grate will be run."
grate --connectionstring="Server=$GR_DEV_DB_PUBLIC_ENDPOINT;Database=$GR_DEV_DB_DBNAME;User Id=$GR_DEV_DB_ADMIN_USERNAME;Password=$GR_DEV_DB_ADMIN_PASSWORD;"
--commandtimeout=3600
--sqlfilesdirectory="./../IdServiceToCDbDataPipeline"
--environment=dev
--noninteractive
--usertokens="GR_DEV_IDSERVICETOCDBDATAPIPELINE_USERNAME=$GR_DEV_IDSERVICETOCDBDATAPIPELINE_USERNAME"
--usertokens="GR_DEV_IDSERVICETOCDBDATAPIPELINE_PASSWORD=$GR_DEV_IDSERVICETOCDBDATAPIPELINE_PASSWORD"
--warnononetimescriptchanges
--transaction
else
if [ -z "$GR_DEV_IDSERVICETOCDBDATAPIPELINE_USERNAME" ]; then
echo "The environment variable GR_DEV_IDSERVICETOCDBDATAPIPELINE_USERNAME is not defined. Grate will not be run."
fi
if [ -z "$GR_DEV_IDSERVICETOCDBDATAPIPELINE_PASSWORD" ]; then
echo "The environment variable GR_DEV_IDSERVICETOCDBDATAPIPELINE_PASSWORD is not defined. Grate will not be run."
fi
if [ -z "$GR_DEV_DB_ADMIN_USERNAME" ]; then
echo "The environment variable GR_DEV_DB_ADMIN_USERNAME is not defined. Grate will not be run."
fi
if [ -z "$GR_DEV_DB_ADMIN_PASSWORD" ]; then
echo "The environment variable GR_DEV_DB_ADMIN_PASSWORD is not defined. Grate will not be run."
fi
if [ -z "$GR_DEV_DB_DBNAME" ]; then
echo "The environment variable GR_DEV_DB_DBNAME is not defined. Grate will not be run."
fi
if [ -z "$GR_DEV_DB_PUBLIC_ENDPOINT" ]; then
echo "The environment variable GR_DEV_DB_PUBLIC_ENDPOINT is not defined. Grate will not be run."
fi
fi
The output of running the shell script is below.
2024-05-09T00:35:06.1697994Z ##[section]Starting: CmdLine
2024-05-09T00:35:06.1702054Z ==============================================================================
2024-05-09T00:35:06.1702179Z Task         : Command line
2024-05-09T00:35:06.1702247Z Description  : Run a command line script using Bash on Linux and macOS and cmd.exe on Windows
2024-05-09T00:35:06.1702383Z Version      : 2.237.1
2024-05-09T00:35:06.1702443Z Author       : Microsoft Corporation
2024-05-09T00:35:06.1702518Z Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks/utility/command-line
2024-05-09T00:35:06.1702634Z ==============================================================================
2024-05-09T00:35:06.3481699Z Generating script.
2024-05-09T00:35:06.3492768Z Script contents:
2024-05-09T00:35:06.3493296Z ./dev.noninteractive.sh
2024-05-09T00:35:06.3493634Z ========================== Starting Command Output ===========================
2024-05-09T00:35:06.3510748Z [command]/usr/bin/bash --noprofile --norc /home/vsts/work/_temp/64302b49-9dc5-49dd-9790-8b4bcbe04eee.sh
2024-05-09T00:35:06.3593266Z All needed environment variables are defined. Grate will be run.
2024-05-09T00:35:06.5799318Z Initializing connections.
2024-05-09T00:35:06.6427618Z Running grate v1.7.2 (build date 05/08/2024 06:03:24) against *** - Dev.
2024-05-09T00:35:06.6450597Z Looking in ./../IdServiceToCDbDataPipeline for scripts to run.
2024-05-09T00:35:06.6460115Z ================================================================================
2024-05-09T00:35:06.6460608Z Setup, Backup, Create/Restore/Drop
2024-05-09T00:35:06.6461581Z ================================================================================
2024-05-09T00:35:07.2916557Z ================================================================================
2024-05-09T00:35:07.2916844Z Grate Structure
2024-05-09T00:35:07.2917035Z ================================================================================
2024-05-09T00:35:07.3487021Z Initializing connections.
2024-05-09T00:35:07.3488446Z Running grate v1.7.2 (build date 05/08/2024 06:03:24) against *** - Dev.
2024-05-09T00:35:07.3494714Z Looking in /tmp/tmpbTIbdb.tmp for scripts to run.
2024-05-09T00:35:07.3495582Z ================================================================================
2024-05-09T00:35:07.3495879Z Setup, Backup, Create/Restore/Drop
2024-05-09T00:35:07.3496085Z ================================================================================
2024-05-09T00:35:07.3496337Z ================================================================================
2024-05-09T00:35:07.3496519Z Grate Structure
2024-05-09T00:35:07.3496725Z ================================================================================
2024-05-09T00:35:07.3500042Z ================================================================================
2024-05-09T00:35:07.3500530Z Versioning
2024-05-09T00:35:07.3501055Z ================================================================================
2024-05-09T00:35:07.3850114Z  Migrating Dev from version 1.7.0 to 1.7.2.
2024-05-09T00:35:07.4546245Z  Versioning Dev database with version 1.7.2.
2024-05-09T00:35:07.4554465Z ================================================================================
2024-05-09T00:35:07.4555135Z Migration Scripts
2024-05-09T00:35:07.4555412Z ================================================================================
2024-05-09T00:35:07.7563996Z Skipping 'BeforeMigration', beforeMigration does not exist.
2024-05-09T00:35:07.8023613Z Skipping 'AlterDatabase', alterDatabase does not exist.
2024-05-09T00:35:07.8027165Z Skipping 'Run After Create Database', runAfterCreateDatabase does not exist.
2024-05-09T00:35:07.8027867Z Skipping 'Run Before Update', runBeforeUp does not exist.
2024-05-09T00:35:07.8028835Z
2024-05-09T00:35:07.8029239Z Looking for Update scripts in "/tmp/tmpbTIbdb.tmp/up". These should be one time only scripts.
2024-05-09T00:35:07.8030127Z --------------------------------------------------------------------------------
2024-05-09T00:35:07.8670789Z  No sql run, either an empty folder, or all files run against destination previously.
2024-05-09T00:35:07.8673956Z --------------------------------------------------------------------------------
2024-05-09T00:35:07.8674472Z
2024-05-09T00:35:07.8675336Z Skipping 'Run First After Update', runFirstAfterUp does not exist.
2024-05-09T00:35:07.8675664Z Skipping 'Functions', functions does not exist.
2024-05-09T00:35:07.8675980Z Skipping 'Views', views does not exist.
2024-05-09T00:35:07.8676222Z Skipping 'Stored Procedures', sprocs does not exist.
2024-05-09T00:35:07.8676478Z Skipping 'Triggers', triggers does not exist.
2024-05-09T00:35:07.8676710Z Skipping 'Indexes', indexes does not exist.
2024-05-09T00:35:07.8677040Z Skipping 'Run after Other Anytime Scripts', runAfterOtherAnyTimeScripts does not exist.
2024-05-09T00:35:07.9095905Z Skipping 'Permissions', permissions does not exist.
2024-05-09T00:35:07.9509680Z Skipping 'AfterMigration', afterMigration does not exist.
2024-05-09T00:35:08.0413422Z
2024-05-09T00:35:08.0413889Z
2024-05-09T00:35:08.0415679Z grate v1.7.2 (build date 05/08/2024 06:03:24) has grated your database (Dev)! You are now at version 1.7.2. All changes and backups can be found at "/home/vsts/.local/share/grate/grate-internal-bootstrap/migrations/Dev/anonymized.database.windows.net_3342/2024-05-09T00_35_07.3480485_00_00".
2024-05-09T00:35:08.0416312Z
2024-05-09T00:35:08.0811480Z Initializing connections.
2024-05-09T00:35:08.0815512Z Running grate v1.7.2 (build date 05/08/2024 06:03:24) against *** - Dev.
2024-05-09T00:35:08.0816083Z Looking in /tmp/tmptdIf0p.tmp for scripts to run.
2024-05-09T00:35:08.0818731Z ================================================================================
2024-05-09T00:35:08.0819899Z Setup, Backup, Create/Restore/Drop
2024-05-09T00:35:08.0820971Z ================================================================================
2024-05-09T00:35:08.0822185Z ================================================================================
2024-05-09T00:35:08.0823118Z Grate Structure
2024-05-09T00:35:08.0824209Z ================================================================================
2024-05-09T00:35:08.0825205Z ================================================================================
2024-05-09T00:35:08.0826149Z Versioning
2024-05-09T00:35:08.0827183Z ================================================================================
2024-05-09T00:35:08.1155727Z  Migrating Dev from version 1.7.0 to 1.7.2.
2024-05-09T00:35:08.1566144Z  Versioning Dev database with version 1.7.2.
2024-05-09T00:35:08.1569773Z ================================================================================
2024-05-09T00:35:08.1570553Z Migration Scripts
2024-05-09T00:35:08.1570872Z ================================================================================
2024-05-09T00:35:08.1902495Z Skipping 'BeforeMigration', beforeMigration does not exist.
2024-05-09T00:35:08.2230407Z Skipping 'AlterDatabase', alterDatabase does not exist.
2024-05-09T00:35:08.2231590Z Skipping 'Run After Create Database', runAfterCreateDatabase does not exist.
2024-05-09T00:35:08.2232937Z Skipping 'Run Before Update', runBeforeUp does not exist.
2024-05-09T00:35:08.2235063Z
2024-05-09T00:35:08.2236255Z Looking for Update scripts in "/tmp/tmptdIf0p.tmp/up". These should be one time only scripts.
2024-05-09T00:35:08.2237370Z --------------------------------------------------------------------------------
2024-05-09T00:35:08.4131959Z  No sql run, either an empty folder, or all files run against destination previously.
2024-05-09T00:35:08.4133337Z --------------------------------------------------------------------------------
2024-05-09T00:35:08.4133971Z
2024-05-09T00:35:08.4134473Z Skipping 'Run First After Update', runFirstAfterUp does not exist.
2024-05-09T00:35:08.4138395Z Skipping 'Functions', functions does not exist.
2024-05-09T00:35:08.4139086Z Skipping 'Views', views does not exist.
2024-05-09T00:35:08.4139422Z Skipping 'Stored Procedures', sprocs does not exist.
2024-05-09T00:35:08.4139711Z Skipping 'Triggers', triggers does not exist.
2024-05-09T00:35:08.4139996Z Skipping 'Indexes', indexes does not exist.
2024-05-09T00:35:08.4140343Z Skipping 'Run after Other Anytime Scripts', runAfterOtherAnyTimeScripts does not exist.
2024-05-09T00:35:08.4558291Z Skipping 'Permissions', permissions does not exist.
2024-05-09T00:35:08.5034408Z Skipping 'AfterMigration', afterMigration does not exist.
2024-05-09T00:35:08.5763252Z
2024-05-09T00:35:08.5763917Z
2024-05-09T00:35:08.5765435Z grate v1.7.2 (build date 05/08/2024 06:03:24) has grated your database (Dev)! You are now at version 1.7.2. All changes and backups can be found at "/home/vsts/.local/share/grate/grate-internal/migrations/Dev/anonymized.database.windows.net_3342/2024-05-09T00_35_08.0815716_00_00".
2024-05-09T00:35:08.5766026Z
2024-05-09T00:35:08.5787823Z Initializing connections.
2024-05-09T00:35:08.5788980Z Running grate v1.7.2 (build date 05/08/2024 06:03:24) against *** - Dev.
2024-05-09T00:35:08.5789605Z Looking in /tmp/tmpaUofAf.tmp for scripts to run.
2024-05-09T00:35:08.5790290Z ================================================================================
2024-05-09T00:35:08.5790557Z Setup, Backup, Create/Restore/Drop
2024-05-09T00:35:08.5790806Z ================================================================================
2024-05-09T00:35:08.5791107Z ================================================================================
2024-05-09T00:35:08.5791540Z Grate Structure
2024-05-09T00:35:08.5791772Z ================================================================================
2024-05-09T00:35:08.5792058Z ================================================================================
2024-05-09T00:35:08.5792274Z Versioning
2024-05-09T00:35:08.5792496Z ================================================================================
2024-05-09T00:35:08.6115897Z  Migrating Dev from version 1.7.0 to 1.7.2.
2024-05-09T00:35:08.6487407Z  Versioning Dev database with version 1.7.2.
2024-05-09T00:35:08.6488015Z ================================================================================
2024-05-09T00:35:08.6488570Z Migration Scripts
2024-05-09T00:35:08.6489397Z ================================================================================
2024-05-09T00:35:08.6815109Z Skipping 'BeforeMigration', beforeMigration does not exist.
2024-05-09T00:35:08.7147064Z Skipping 'AlterDatabase', alterDatabase does not exist.
2024-05-09T00:35:08.7147883Z Skipping 'Run After Create Database', runAfterCreateDatabase does not exist.
2024-05-09T00:35:08.7148398Z Skipping 'Run Before Update', runBeforeUp does not exist.
2024-05-09T00:35:08.7148808Z
2024-05-09T00:35:08.7149283Z Looking for Update scripts in "/tmp/tmpaUofAf.tmp/up". These should be one time only scripts.
2024-05-09T00:35:08.7166944Z --------------------------------------------------------------------------------
2024-05-09T00:35:08.7827069Z  No sql run, either an empty folder, or all files run against destination previously.
2024-05-09T00:35:08.7827741Z --------------------------------------------------------------------------------
2024-05-09T00:35:08.7828069Z
2024-05-09T00:35:08.7828366Z Skipping 'Run First After Update', runFirstAfterUp does not exist.
2024-05-09T00:35:08.7828677Z Skipping 'Functions', functions does not exist.
2024-05-09T00:35:08.7829280Z Skipping 'Views', views does not exist.
2024-05-09T00:35:08.7829558Z Skipping 'Stored Procedures', sprocs does not exist.
2024-05-09T00:35:08.7829971Z Skipping 'Triggers', triggers does not exist.
2024-05-09T00:35:08.7830238Z Skipping 'Indexes', indexes does not exist.
2024-05-09T00:35:08.7830600Z Skipping 'Run after Other Anytime Scripts', runAfterOtherAnyTimeScripts does not exist.
2024-05-09T00:35:08.8241282Z Skipping 'Permissions', permissions does not exist.
2024-05-09T00:35:08.8654989Z Skipping 'AfterMigration', afterMigration does not exist.
2024-05-09T00:35:08.9383891Z
2024-05-09T00:35:08.9384525Z
2024-05-09T00:35:08.9386030Z grate v1.7.2 (build date 05/08/2024 06:03:24) has grated your database (Dev)! You are now at version 1.7.2. All changes and backups can be found at "/home/vsts/.local/share/grate/grate-internal-bootstrap/migrations/Dev/anonymized.database.windows.net_3342/2024-05-09T00_35_08.5784940_00_00".
2024-05-09T00:35:08.9388693Z
2024-05-09T00:35:08.9393858Z Initializing connections.
2024-05-09T00:35:08.9397305Z Running grate v1.7.2 (build date 05/08/2024 06:03:24) against *** - Dev.
2024-05-09T00:35:08.9397586Z Looking in /tmp/tmpIw2R3X.tmp for scripts to run.
2024-05-09T00:35:08.9397861Z ================================================================================
2024-05-09T00:35:08.9398106Z Setup, Backup, Create/Restore/Drop
2024-05-09T00:35:08.9398339Z ================================================================================
2024-05-09T00:35:08.9398630Z ================================================================================
2024-05-09T00:35:08.9398836Z Grate Structure
2024-05-09T00:35:08.9399071Z ================================================================================
2024-05-09T00:35:08.9399359Z ================================================================================
2024-05-09T00:35:08.9399559Z Versioning
2024-05-09T00:35:08.9399793Z ================================================================================
2024-05-09T00:35:08.9728446Z  Migrating Dev from version 1.7.0 to 1.7.2.
2024-05-09T00:35:09.0095288Z  Versioning Dev database with version 1.7.2.
2024-05-09T00:35:09.0095647Z ================================================================================
2024-05-09T00:35:09.0095920Z Migration Scripts
2024-05-09T00:35:09.0096159Z ================================================================================
2024-05-09T00:35:09.0426488Z Skipping 'BeforeMigration', beforeMigration does not exist.
2024-05-09T00:35:09.0763311Z Skipping 'AlterDatabase', alterDatabase does not exist.
2024-05-09T00:35:09.0767384Z Skipping 'Run After Create Database', runAfterCreateDatabase does not exist.
2024-05-09T00:35:09.0767748Z Skipping 'Run Before Update', runBeforeUp does not exist.
2024-05-09T00:35:09.0771897Z
2024-05-09T00:35:09.0772145Z Looking for Update scripts in "/tmp/tmpIw2R3X.tmp/up". These should be one time only scripts.
2024-05-09T00:35:09.0773109Z --------------------------------------------------------------------------------
2024-05-09T00:35:09.1486255Z  No sql run, either an empty folder, or all files run against destination previously.
2024-05-09T00:35:09.1487237Z --------------------------------------------------------------------------------
2024-05-09T00:35:09.1487967Z
2024-05-09T00:35:09.1488446Z Skipping 'Run First After Update', runFirstAfterUp does not exist.
2024-05-09T00:35:09.1488766Z Skipping 'Functions', functions does not exist.
2024-05-09T00:35:09.1489043Z Skipping 'Views', views does not exist.
2024-05-09T00:35:09.1489321Z Skipping 'Stored Procedures', sprocs does not exist.
2024-05-09T00:35:09.1489616Z Skipping 'Triggers', triggers does not exist.
2024-05-09T00:35:09.1489880Z Skipping 'Indexes', indexes does not exist.
2024-05-09T00:35:09.1490592Z Skipping 'Run after Other Anytime Scripts', runAfterOtherAnyTimeScripts does not exist.
2024-05-09T00:35:09.1898875Z Skipping 'Permissions', permissions does not exist.
2024-05-09T00:35:09.2318244Z Skipping 'AfterMigration', afterMigration does not exist.
2024-05-09T00:35:09.3247863Z
2024-05-09T00:35:09.3248133Z
2024-05-09T00:35:09.3249349Z grate v1.7.2 (build date 05/08/2024 06:03:24) has grated your database (Dev)! You are now at version 1.7.2. All changes and backups can be found at "/home/vsts/.local/share/grate/grate-internal/migrations/Dev/anonymized.database.windows.net_3342/2024-05-09T00_35_08.9390750_00_00".
2024-05-09T00:35:09.3249821Z
2024-05-09T00:35:09.3250035Z ================================================================================
2024-05-09T00:35:09.3250254Z Versioning
2024-05-09T00:35:09.3250466Z ================================================================================
2024-05-09T00:35:09.3594989Z  Migrating Dev from version <IP_ADDRESS> to <IP_ADDRESS>.
2024-05-09T00:35:09.3977345Z  Versioning Dev database with version <IP_ADDRESS>.
2024-05-09T00:35:09.3977643Z ================================================================================
2024-05-09T00:35:09.3977875Z Migration Scripts
2024-05-09T00:35:09.3978090Z ================================================================================
2024-05-09T00:35:09.4327401Z Skipping 'BeforeMigration', beforeMigration does not exist.
2024-05-09T00:35:09.4851738Z Skipping 'AlterDatabase', alterDatabase does not exist.
2024-05-09T00:35:09.4852647Z Skipping 'Run After Create Database', runAfterCreateDatabase does not exist.
2024-05-09T00:35:09.4853153Z Skipping 'Run Before Update', runBeforeUp does not exist.
2024-05-09T00:35:09.4855824Z
2024-05-09T00:35:09.4856198Z Looking for Update scripts in "./../IdServiceToCDbDataPipeline/up". These should be one time only scripts.
2024-05-09T00:35:09.4859020Z --------------------------------------------------------------------------------
2024-05-09T00:35:09.6326200Z   Running '000_IdSToCDbDP_CreateLogin.dev.env.sql'.
2024-05-09T00:35:09.7626165Z   Running '001_IdSToCDbDP_CreateUser.dev.env.sql'.
2024-05-09T00:35:09.8683462Z   Running '002_IdSToCDbDP_AlterRoles.dev.env.sql'.
2024-05-09T00:35:09.9691447Z   Running '003_IdSToCDbDP_InsightlySchema.sql'.
2024-05-09T00:35:10.0682724Z   Running '004_IdSToCDbDP_45.sql'.
2024-05-09T00:35:10.1773151Z --------------------------------------------------------------------------------
2024-05-09T00:35:10.1773505Z
2024-05-09T00:35:10.1776647Z Skipping 'Run First After Update', runFirstAfterUp does not exist.
2024-05-09T00:35:10.1780641Z Skipping 'Functions', functions does not exist.
2024-05-09T00:35:10.1781345Z Skipping 'Views', views does not exist.
2024-05-09T00:35:10.1781513Z
2024-05-09T00:35:10.1783933Z Looking for Stored Procedures scripts in "./../IdServiceToCDbDataPipeline/sprocs".
2024-05-09T00:35:10.1784315Z --------------------------------------------------------------------------------
2024-05-09T00:35:10.2114572Z   Running 'IdSToCDbDP_AdvancedMD_sp_GetPatientsChangedOrWithoutROMTechId.sql'.
2024-05-09T00:35:10.3241701Z   Running 'IdSToCDbDP_Okta_sp_GetUsersChangedOrWithoutROMTechId.sql'.
2024-05-09T00:35:10.3932469Z --------------------------------------------------------------------------------
2024-05-09T00:35:10.3932780Z
2024-05-09T00:35:10.3933297Z Skipping 'Triggers', triggers does not exist.
2024-05-09T00:35:10.3933675Z Skipping 'Indexes', indexes does not exist.
2024-05-09T00:35:10.3934115Z Skipping 'Run after Other Anytime Scripts', runAfterOtherAnyTimeScripts does not exist.
2024-05-09T00:35:10.4366341Z
2024-05-09T00:35:10.4367165Z Looking for Permissions scripts in "./../IdServiceToCDbDataPipeline/permissions". These scripts will run every time.
2024-05-09T00:35:10.4367970Z --------------------------------------------------------------------------------
2024-05-09T00:35:10.4797510Z   Running 'IdSToCDbDP_AdvancedMD_execute_perms.dev.env.sql'.
2024-05-09T00:36:10.3841251Z IdSToCDbDP_AdvancedMD_execute_perms.dev.env.sql: Cannot find the user 'dev_IdServiceToCDbDataPipeline', because it does not exist or you do not have permission.
2024-05-09T00:36:10.8047242Z Skipping 'AfterMigration', afterMigration does not exist.
2024-05-09T00:36:10.8142263Z An error occurred: Migration failed due to the following errors:
2024-05-09T00:36:10.8142685Z
2024-05-09T00:36:10.8144547Z Permissions ("permissions"):
2024-05-09T00:36:10.8145119Z --------------------------------------------------------------------------------
2024-05-09T00:36:10.8209122Z IdSToCDbDP_AdvancedMD_execute_perms.dev.env.sql: Cannot find the user 'dev_IdServiceToCDbDataPipeline', because it does not exist or you do not have permission.
2024-05-09T00:36:10.8212576Z (0,0): grant execute on schema::AdvancedMD to {{GR_DEV_IDSERVICETOCDBDATAPIPELINE_USERNAME}}
2024-05-09T00:36:10.8213034Z       ^ Cannot find the user 'dev_IdServiceToCDbDataPipeline', because it does not exist or you do not have permission.
2024-05-09T00:36:10.8305798Z
2024-05-09T00:36:10.8355171Z ##[error]Bash exited with code '1'.
2024-05-09T00:36:10.8379535Z ##[section]Finishing: CmdLine
The permissions script does run, but appears not to have access to the user created in the "Update" scripts. The problem manifests itself slightly differently in 1.7.2, with the permissions script timing out with the message "Execution Timeout Expired.  The timeout period elapsed prior to completion of the operation or the server is not responding." after about 30 seconds. Reverting to 1.6.2 resolves the issue.
If there's anything else you need, please let me know, and please understand that you have my (and likely many others') unrelenting gratitude for your work on RoundhousE and grate.
Thanks,
Dó
From: Erik A. Brandstadmoen @.>
Sent: Thursday, May 9, 2024 6:26 AM
To: erikbra/grate @.>
Cc: DonalLOConnor @.>; Mention @.>
Subject: Re: [erikbra/grate] Migrations keep looping / timeout on permissions scripts on v1.7.0 (Issue #508)
Thanks for chiming in, @DonalLOConnorhttps://github.com/DonalLOConnor !
To help me understand better, could you please give me some details about how you run grate? Is it the normal command-line version? Or docker? Are you also on Windows?
Could you please provide me with the command-line you use, and, also the connection string (without the secrets of course). I'm very interested in which type of authentication mechanism you are using. Is it Azure AD (aka Microsoft Entra) auth, or SQL server username and password (ref e.g. https://learn.microsoft.com/en-us/sql/connect/ado-net/sql/azure-active-directory-authentication?view=sql-server-ver16#setting-microsoft-entra-authentication)
Additionally, after you see the output from the two "internal grate migrations", which are expected, does your permission scripts not run too? Does it hang? Timeout? What happens? Could you please provide me with a full output of the run (you can of course anonymise it if you have anything you don't want to share with the whole world)? It will make it easier to understand what happens, and solve the problem.
Is the problem there in grate 1.7.2 also (released yesterday)?
—
Reply to this email directly, view it on GitHubhttps://github.com/erikbra/grate/issues/508#issuecomment-2102392218, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AEB45XBW4CY3T256WZKL3GDZBNFMPAVCNFSM6AAAAABG7TENWSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMBSGM4TEMRRHA.
You are receiving this because you were mentioned.Message ID<EMAIL_ADDRESS>

Thanks a lot for this output, @DonalLOConnor !
Explanation
This output is, although a bit confusing, actually just as expected. For grate 1.7+, I run some internal grate migrations (two) on the beginning of each migration, to make sure the tables used for storing the history of the migrations are up-to-date.
The intent was that this output shouldn't show, as it is of no interest to anyone but the grate developers, but due to a bug, the logs hit the end-users as well. This is fixed in release 1.7.3.
To explain, the migrations after these lines:
2024-05-09T00:35:07.3494714Z Looking in /tmp/tmpbTIbdb.tmp for scripts to run.
and
2024-05-09T00:35:08.0816083Z Looking in /tmp/tmptdIf0p.tmp for scripts to run.
are the internal migrations. Your actual migration starts on top, from:
2024-05-09T00:35:06.6427618Z Running grate v1.7.2 (build date 05/08/2024 06:03:24) against *** - Dev.
up until the line:
2024-05-09T00:35:07.2916844Z Grate Structure
and, then again from:
2024-05-09T00:35:09.3250254Z Versioning
all between these lines are the internal grate migrations, and this output is removed in 1.7.3.
TLDR;
Now, to your actual issue, it's difficult to know whether this is a grate bug or not. It says is cannot find the user dev_IdServiceToCDbDataPipeline. Are you absolutely sure that is the name of the user created in the scripts where you create the user? Do you use tokens in the "create user" scripts? Could it be that the casing is different?
Could you please retry the scripts on grate 1.7.3, where we get rid of all the confusing logging, and see if you still think there is a grate error?
Could you please try grate 1.7.3 too, and see if you still experience any hangs/strange behaviour, @kimjamia ?

I created a new empty db and ran grate locally and waited until it exited (previously I killed it after I saw it ran multiple migrations). The DB is an Azure SQL Database and I'm using access token auth. It didn't time out, though. The timeout was happening in our Azure DevOps pipeline.
Basically it's running 5 sets of migrations here:

Create the grate schema and the tables.
Run them again (except 01_create_schema_grate.ENV.GrateInternalBoostrap).
Run the migrations that add a status column and unicode support.
Run them again.
Run my migrations.

The tables it created:

The output:
PS > dotnet grate --connectionstring "Server=***.database.windows.net;Database=kimtest2" --accesstoken $sqltoken -f .\scripts\
Initializing connections.
Running grate v1.7.0 (build date 04/29/2024 17:52:09) against ***.database.windows.net - kimtest2.
Looking in .\scripts\ for scripts to run.
Please press enter when ready to kick...

================================================================================
Setup, Backup, Create/Restore/Drop
================================================================================
================================================================================
Grate Structure
================================================================================
Initializing connections.
Running grate v1.7.0 (build date 04/29/2024 17:52:09) against ***.database.windows.net - kimtest2.
Looking in C:\Users\***\AppData\Local\Temp\tmpd2owzz.tmp for scripts to run.
================================================================================
Setup, Backup, Create/Restore/Drop
================================================================================
================================================================================
Grate Structure
================================================================================
================================================================================
Versioning
================================================================================
 Migrating kimtest2 from version <IP_ADDRESS> to 1.7.0.
 Versioning kimtest2 database with version 1.7.0.
================================================================================
Migration Scripts
================================================================================
Skipping 'BeforeMigration', beforeMigration does not exist.
Skipping 'AlterDatabase', alterDatabase does not exist.
Skipping 'Run After Create Database', runAfterCreateDatabase does not exist.
Skipping 'Run Before Update', runBeforeUp does not exist.

Looking for Update scripts in "C:\Users\***\AppData\Local\Temp\tmpd2owzz.tmp\up". These should be one time only scripts.
--------------------------------------------------------------------------------
  Running 'grate-internal/01_create_schema_grate.ENV.GrateInternalBoostrap-a61456d0-e00a-4933-b692-c6a5d7d51539.sql'.
  Running 'grate-internal/02_create_scripts_run_table.sql'.
  Running 'grate-internal/03_create_scripts_run_errors_table.sql'.
  Running 'grate-internal/04_create_version_table.sql'.
--------------------------------------------------------------------------------

Skipping 'Run First After Update', runFirstAfterUp does not exist.
Skipping 'Functions', functions does not exist.
Skipping 'Views', views does not exist.
Skipping 'Stored Procedures', sprocs does not exist.
Skipping 'Triggers', triggers does not exist.
Skipping 'Indexes', indexes does not exist.
Skipping 'Run after Other Anytime Scripts', runAfterOtherAnyTimeScripts does not exist.
Skipping 'Permissions', permissions does not exist.
Skipping 'AfterMigration', afterMigration does not exist.

grate v1.7.0 (build date 04/29/2024 17:52:09) has grated your database (kimtest2)! You are now at version 1.7.0. All changes and backups can be found at "C:\Users\***\gratetest\output\migrations\kimtest2\***.database.windows.net\2024-05-13T11_34_19.7164588_03_00".

Initializing connections.
Running grate v1.7.0 (build date 04/29/2024 17:52:09) against ***.database.windows.net - kimtest2.
Looking in C:\Users\***\AppData\Local\Temp\tmpqxg4hs.tmp for scripts to run.
================================================================================
Setup, Backup, Create/Restore/Drop
================================================================================
================================================================================
Grate Structure
================================================================================
================================================================================
Versioning
================================================================================
 Migrating kimtest2 from version <IP_ADDRESS> to 1.7.0.
 Versioning kimtest2 database with version 1.7.0.
================================================================================
Migration Scripts
================================================================================
Skipping 'BeforeMigration', beforeMigration does not exist.
Skipping 'AlterDatabase', alterDatabase does not exist.
Skipping 'Run After Create Database', runAfterCreateDatabase does not exist.
Skipping 'Run Before Update', runBeforeUp does not exist.

Looking for Update scripts in "C:\Users\***\AppData\Local\Temp\tmpqxg4hs.tmp\up". These should be one time only scripts.
--------------------------------------------------------------------------------
  Running '02_create_scripts_run_table.sql'.
  Running '03_create_scripts_run_errors_table.sql'.
  Running '04_create_version_table.sql'.
--------------------------------------------------------------------------------

Skipping 'Run First After Update', runFirstAfterUp does not exist.
Skipping 'Functions', functions does not exist.
Skipping 'Views', views does not exist.
Skipping 'Stored Procedures', sprocs does not exist.
Skipping 'Triggers', triggers does not exist.
Skipping 'Indexes', indexes does not exist.
Skipping 'Run after Other Anytime Scripts', runAfterOtherAnyTimeScripts does not exist.
Skipping 'Permissions', permissions does not exist.
Skipping 'AfterMigration', afterMigration does not exist.

grate v1.7.0 (build date 04/29/2024 17:52:09) has grated your database (kimtest2)! You are now at version 1.7.0. All changes and backups can be found at "C:\Users\***\gratetest\output\migrations\kimtest2\***.database.windows.net\2024-05-13T11_34_21.6520724_03_00".

Initializing connections.
Running grate v1.7.0 (build date 04/29/2024 17:52:09) against ***.database.windows.net - kimtest2.
Looking in C:\Users\***\AppData\Local\Temp\tmpaasrkt.tmp for scripts to run.
================================================================================
Setup, Backup, Create/Restore/Drop
================================================================================
================================================================================
Grate Structure
================================================================================
================================================================================
Versioning
================================================================================
 Migrating kimtest2 from version 1.7.0 to 1.7.0.
 Versioning kimtest2 database with version 1.7.0.
================================================================================
Migration Scripts
================================================================================
Skipping 'BeforeMigration', beforeMigration does not exist.
Skipping 'AlterDatabase', alterDatabase does not exist.
Skipping 'Run After Create Database', runAfterCreateDatabase does not exist.
Skipping 'Run Before Update', runBeforeUp does not exist.

Looking for Update scripts in "C:\Users\***\AppData\Local\Temp\tmpaasrkt.tmp\up". These should be one time only scripts.
--------------------------------------------------------------------------------
  Running 'grate-internal/01_version_add_status_column.sql'.
  Running 'grate-internal/02_scripts_run_unicode_support.sql'.
  Running 'grate-internal/03_scripts_run_errors_unicode_support.sql'.
--------------------------------------------------------------------------------

Skipping 'Run First After Update', runFirstAfterUp does not exist.
Skipping 'Functions', functions does not exist.
Skipping 'Views', views does not exist.
Skipping 'Stored Procedures', sprocs does not exist.
Skipping 'Triggers', triggers does not exist.
Skipping 'Indexes', indexes does not exist.
Skipping 'Run after Other Anytime Scripts', runAfterOtherAnyTimeScripts does not exist.
Skipping 'Permissions', permissions does not exist.
Skipping 'AfterMigration', afterMigration does not exist.

grate v1.7.0 (build date 04/29/2024 17:52:09) has grated your database (kimtest2)! You are now at version 1.7.0. All changes and backups can be found at "C:\Users\***\gratetest\output\migrations\kimtest2\***.database.windows.net\2024-05-13T11_34_22.8775005_03_00".

Initializing connections.
Running grate v1.7.0 (build date 04/29/2024 17:52:09) against ***.database.windows.net - kimtest2.
Looking in C:\Users\***\AppData\Local\Temp\tmp2d2iue.tmp for scripts to run.
================================================================================
Setup, Backup, Create/Restore/Drop
================================================================================
================================================================================
Grate Structure
================================================================================
================================================================================
Versioning
================================================================================
 Migrating kimtest2 from version 1.7.0 to 1.7.0.
 Versioning kimtest2 database with version 1.7.0.
================================================================================
Migration Scripts
================================================================================
Skipping 'BeforeMigration', beforeMigration does not exist.
Skipping 'AlterDatabase', alterDatabase does not exist.
Skipping 'Run After Create Database', runAfterCreateDatabase does not exist.
Skipping 'Run Before Update', runBeforeUp does not exist.

Looking for Update scripts in "C:\Users\***\AppData\Local\Temp\tmp2d2iue.tmp\up". These should be one time only scripts.
--------------------------------------------------------------------------------
  Running '01_version_add_status_column.sql'.
  Running '02_scripts_run_unicode_support.sql'.
  Running '03_scripts_run_errors_unicode_support.sql'.
--------------------------------------------------------------------------------

Skipping 'Run First After Update', runFirstAfterUp does not exist.
Skipping 'Functions', functions does not exist.
Skipping 'Views', views does not exist.
Skipping 'Stored Procedures', sprocs does not exist.
Skipping 'Triggers', triggers does not exist.
Skipping 'Indexes', indexes does not exist.
Skipping 'Run after Other Anytime Scripts', runAfterOtherAnyTimeScripts does not exist.
Skipping 'Permissions', permissions does not exist.
Skipping 'AfterMigration', afterMigration does not exist.

grate v1.7.0 (build date 04/29/2024 17:52:09) has grated your database (kimtest2)! You are now at version 1.7.0. All changes and backups can be found at "C:\Users\***\gratetest\output\migrations\kimtest2\***.database.windows.net\2024-05-13T11_34_24.2423268_03_00".

================================================================================
Versioning
================================================================================
 Migrating kimtest2 from version <IP_ADDRESS> to <IP_ADDRESS>.
 Versioning kimtest2 database with version <IP_ADDRESS>.
================================================================================
Migration Scripts
================================================================================
Skipping 'BeforeMigration', beforeMigration does not exist.
Skipping 'AlterDatabase', alterDatabase does not exist.
Skipping 'Run After Create Database', runAfterCreateDatabase does not exist.
Skipping 'Run Before Update', runBeforeUp does not exist.
Skipping 'Update', up does not exist.
Skipping 'Run First After Update', runFirstAfterUp does not exist.
Skipping 'Functions', functions does not exist.
Skipping 'Views', views does not exist.
Skipping 'Stored Procedures', sprocs does not exist.
Skipping 'Triggers', triggers does not exist.
Skipping 'Indexes', indexes does not exist.
Skipping 'Run after Other Anytime Scripts', runAfterOtherAnyTimeScripts does not exist.

Looking for Permissions scripts in ".\scripts\permissions". These scripts will run every time.
--------------------------------------------------------------------------------
  Running 'test.sql'.
--------------------------------------------------------------------------------

Skipping 'AfterMigration', afterMigration does not exist.

grate v1.7.0 (build date 04/29/2024 17:52:09) has grated your database (kimtest2)! You are now at version <IP_ADDRESS>. All changes and backups can be found at "C:\Users\***\AppData\Local\grate\migrations\kimtest2\***.database.windows.net\2024-05-13T11_34_18.9493433_03_00".

@kimjamia, thank you. This makes perfect sense, and this is as expected (even though you shouldn't be exposed to it, as I planned to turn the logging off, which will be fixed in next release).
The tables that are called:

GrateScriptsRun
GrateScriptsErrors
GrateVersion
are used for versioning the grate tables themselves (i.e. ScriptsRun, ScriptsRunErrors, and GrateVersion).

I thought about putting the "internal migrations" in the "normal" grate tables, but didn't want to interfere with people's "real" migrations.
So, as the internal folder structure for Grate evolves, the migrations of those tables will be in the Grate* tables.
There are two sets of migrations for the internal tables, one to update these tables above, and one that updates the Grate* tables that are used for user migrations. They should both evolve.
And, there is a bootstrapping the first time, to create the tables, and there is a migration that adds a column and changes some columns.
So two times two internal migrations makes sense for the first 1.7 run

Aha, @DonalLOConnor - that explains it. Great to hear you were able to solve your particular issue too! Shell escaping in Azure Devops tasks are a pain. I've experienced just having to give up some times, when you e.g. want to actually provide a string in quotes to a task (e.g. a list of tags to build a docker image from using dotnet publish to docker).
Anyway, I'm thinking, if you see any things that could be better if we make a custom-built native Devops-task for grate, please let me know. My personal experience is that some of the existing devops tasks out there are actually less convenient to work with that just using a shell, but your experience may differ. If you have any good ideas, please feel free to file an issue, or start a discussion around this!
I'm also thinking a bit around how to more easily configure grate using the environment (e.g. environment variables). I don't think System.CommandLine, which I'm using for parsing the arguments is going to be released in the current form at all. I want to wait a bit, and see if there is any movement around native command-argument parsing in the dotnet runtime coming in .NET 9 (coming in October). If not, I might rewrite it to use plain, old .NET Configuration providers. We'll see :)

@kimjamia - thanks a lot for you simple repro. That helped me a lot being able to understand what happened. Given my explanation, do you still think we have a bug, or was it just difficult to understand what happened?

Closing this, as I think we solved it, and I'm getting no more feedback.


Fix: text overflow for Question in card

Fix #86

thanks for the visit bro! 😁

I would like to work more on it @jerommiole

Feel free to pull request bro. 😀


Better docs for Product Attributes

Product Attributes need their own documentation and best practices section.
What I would like to see there:

info that I can and it would be good idea do have same slug (ex. color) if I have few attributes with same purpose for different product types
it would be good to mention that  Attribute Value slugs don't have to be unique system vide

I think it would be great to have a guide on how to use attributes programmatically. Example: how to implement a system where the code is able to identify attributes responsible for product color.


Multiples regions through Docker Windows 10.

Hello,
I have read the guide https://nominatim.org/release-docs/develop/admin/Advanced-Installations/#importing-multiple-regions, and don't understand how do I implement it through Docker on Windows 10. Is there some more detailed guide I could miss?

Hi,
this feature is not supported from the docker image right away right now. There is an open PR (https://github.com/mediagis/nominatim-docker/pull/186) which needs major refactoring. Right now I would advise you to use the region that has all the data you need, sorry.


Update branch alias

2.4.0 is released now. That's why the branch alias in the master branch should be changed to 2.4.x

thanks


Update blame help

Summary
Updated help for dotnet test --blame and the related options.

Committed both, thanks!


markdown lint on CHANGELOG.md and examples/READE.md

new lints after #1783, fixes current CI break

bors r+


Add latest tag to most recently uploaded Docker image

The recently released v1.16-live Docker image lacks the latest image tag. The latest tag should be applied to the most recently uploaded Docker image.

@tomholford is reporting an issue with v1.16-live Docker image, so I've pushed tloncorp/vere:latest as an alias for tloncorp/urbit:v1.15 while I look into this further.

This is resolved via #123.