SQL Server Profiler Fails to create new session.

Issue Type: Bug
The SQL Server Profiler Extension fails to create a new session. This just appears to be a limit of how the sessions are being created due to a specified buffer size that ends up being smaller than the minimum then what Azure SQL DB or even an On Prem server needs based on the installed RAM. A high amount of RAM installed will cause the problem.
Steps to Reproduce.

Open Azure Data Studio.
Make sure to have the Profiler v0.11.0 installed or the Admin Pack for SQL Server 0.0.2.
Once installed connect to an Azure Database or any other type of SQL Server with a large amount of memory.
a.	(Any tier or server with 256GB of memory should do)
Press ALT+P or right click on the instance/AzureDB and select Launch Profiler.
Chose Standard_Azure if on AzureDB or Standard_OnPrem/TSQL_OnPrem if on a on premise server.
Here the error "Failed to start session: An exception occurred while executing a Transact-SQL statement or batch." will occur.

If you run the code it is trying to do in Azure Data Studio 1.34.0 or in SSMS 18.10 in a query window it will still cause the error similar to "The specified buffer size is less than the minimum size. The minimum allowed size is 16986060 bytes". (Code below). Either removing the MAX_MEMORY option or setting it to a level higher than that 16986060 bytes in this example it will work fine. So setting to 16,588 KB works. What would seem to need to be done here is either remove the options, or give the user an option to select the size of the buffer allowed to prevent this from happening as that buffer size differs depending on how much RAM is on the system itself. That buffer is not based on what SQL is using or what is allocated to SQL with a max memory setting.
CREATE EVENT SESSION [ADS_Standard_Azure] ON DATABASE ADD EVENT sqlserver.attention (ACTION(package0.event_sequence, sqlserver.client_app_name, sqlserver.client_pid, sqlserver.database_id, sqlserver.username, sqlserver.query_hash, sqlserver.session_id) WHERE ([package0].[equal_boolean]([sqlserver].[is_system], (0))))
    ,ADD EVENT sqlserver.existing_connection (SET collect_options_text = (1) ACTION(package0.event_sequence, sqlserver.client_app_name, sqlserver.client_pid, sqlserver.username, sqlserver.session_id))
    ,ADD EVENT sqlserver.LOGIN (SET collect_options_text = (1) ACTION(package0.event_sequence, sqlserver.client_app_name, sqlserver.client_pid, sqlserver.username, sqlserver.session_id))
    ,ADD EVENT sqlserver.logout (ACTION(package0.event_sequence, sqlserver.client_app_name, sqlserver.client_pid, sqlserver.username, sqlserver.session_id))
    ,ADD EVENT sqlserver.rpc_completed (ACTION(package0.event_sequence, sqlserver.client_app_name, sqlserver.client_pid, sqlserver.database_id, sqlserver.username, sqlserver.query_hash, sqlserver.session_id) WHERE ([package0].[equal_boolean]([sqlserver].[is_system], (0))))
    ,ADD EVENT sqlserver.sql_batch_completed (ACTION(package0.event_sequence, sqlserver.client_app_name, sqlserver.client_pid, sqlserver.database_id, sqlserver.username, sqlserver.query_hash, sqlserver.session_id) WHERE ([package0].[equal_boolean]([sqlserver].[is_system], (0))))
    ,ADD EVENT sqlserver.sql_batch_starting (ACTION(package0.event_sequence, sqlserver.client_app_name, sqlserver.client_pid, sqlserver.database_id, sqlserver.username, sqlserver.query_hash, sqlserver.session_id) WHERE ([package0].[equal_boolean]([sqlserver].[is_system], (0)))) ADD TARGET package0.ring_buffer (
    SET max_events_limit = (1000)
    ,max_memory = (51200)
    )
    WITH (
            MAX_MEMORY = 8192 KB
            ,EVENT_RETENTION_MODE = ALLOW_SINGLE_EVENT_LOSS
            ,MAX_DISPATCH_LATENCY = 5 SECONDS
            ,MAX_EVENT_SIZE = 0 KB
            ,MEMORY_PARTITION_MODE = PER_CPU
            ,TRACK_CAUSALITY = ON
            ,STARTUP_STATE = OFF
            )

Extension version: 0.11.0
Azure Data Studio version: azuredatastudio 1.34.0 (83a4316cf89cbf961a0f09a72c861379cfd01fa9, 2021-12-15T20:47:33.863Z)
OS version: Windows_NT x64 10.0.19044
Restricted Mode: No

System Info

Item
Value

CPUs
Intel(R) Core(TM) i7-6820HQ CPU @ 2.70GHz (8 x 2712)

GPU Status
2d_canvas: enabledgpu_compositing: enabledmultiple_raster_threads: enabled_onoop_rasterization: enabledopengl: enabled_onrasterization: enabledskia_renderer: enabled_onvideo_decode: enabledvulkan: disabled_offwebgl: enabledwebgl2: enabled

Load (avg)
undefined

Memory (System)
31.89GB (12.46GB free)

Process Argv

Screen Reader
no

VM
0%

i have faced this issue today. when i press Alt-P to open SQL Server Profiler
SQL Server Version: 16.0.1000.6
Azure Data Studio: 1.43.0
SQL Server Profiler :  0.12.2
An exception occurred while executing a Transact-SQL statement or batch. ---> The specified buffer size is less than the minimum size. The minimum allowed size is 11759580 bytes.


grammar issue

Issue in help/edge/fundamentals/installing-the-sdk.md
"The AEP web SDK is available on a CDN for your to use."
you not your.

Fixed in PR 737


Fix build on gentoo linux

https://github.com/PirateNetwork/PirateWallet-Lite/issues/30 is more urgent than this.


Backport v<IP_ADDRESS>

It backports #1467 to v0.10.32.
The issue was considered to be unrelated in v0.10.32 (as described in #1467), but actually, it affects when Embulk is called via EmbulkEmbed, not from a command-line.

Thanks!


Can't be used with System.Text.Json source generation

While working on a project using ActivityStreams I thought I'd try using the new System.Text.Json source generation.
Unfortunately there's an issue because the converter types are internal.
[JsonSerializable(typeof(Person))]
public partial class ActivityStreamJsonSerializerContext : JsonSerializerContext {}

Produces the following compiler errors.

error CS0122: 'LinkConverter' is inaccessible due to its protection level
error CS0122: 'EndpointsOrLinkConverter' is inaccessible due to its protection level
error CS0122: 'ObjectOrLinkConverter' is inaccessible due to its protection level
error CS0122: 'ImageOrLinkConverter' is inaccessible due to its protection level
error CS0122: 'CollectionPageOrLinkConverter' is inaccessible due to its protection level
error CS0122: 'TermDefinitionConverter' is inaccessible due to its protection level
error CS0122: 'XMLTimeSpanConverter' is inaccessible due to its protection level

Could the converters please be changed to public?

The change should be available in version 0.2.1.
Please don't hesitate to reach out if you have any other problems. 😊


com.intellij.diagnostic.PluginException: Cannot init toolwindow net.cakebuild.toolwindow.CakeTasksWindowFactory@3dd1d8b2 [Plugin: net.cakebuild.cakerider]

Plugin version: 3.2.0
IDE version: 2022.1 (RD-221.5080.236)
Operating system: Windows 10
Exception: com.intellij.diagnostic.PluginException: Cannot init toolwindow net.cakebuild.toolwindow.CakeTasksWindowFactory@3dd1d8b2 [Plugin: net.cakebuild.cakerider]
Stacktrace:

StackTrace too long. Please copy the StackTrace here manually.

This happens since Rider version 2022.1, when multiple plugins are installed that provide a ToolWindow with the same name.
Specifically, this error occurs, if the CakeIntegration-Plugin is installed alongside of Cake for Rider, as both implement a ToolWindow named 'Cake Tasks'.
Closing this as a duplicate of https://github.com/cake-build/cake-rider/issues/239.


Allow ES6 shorthand methods and properties

Since IE11 is the only browser that wouldn't support them, and support is being dropped for it anyway allow developers to use ECMAScript6 shorthand methods and properties:
({
   doInit (component, event, helper) {
   }
})

:+1 this would greatly increase readability of helper files by eliminating visual noise


Bump Cassandra to 2.0.16.

The current version is 2.0.0, and has some known issues.

LGTM

This addresses and issue where Cassandra fails periodically on some AWS instance types.


feat: add mdx-analyzer support

Hi, here is a draft pull request to add mdx-analyzer support. :purple_heart:
See also:

https://github.com/williamboman/mason-lspconfig.nvim/pull/268
https://mason-registry.dev/registry/list#mdx-analyzer

Thanks @glepnir ! :relaxed:


Introduce qualifications and employment checks

We chose to start with qualification and employment checks as they are common to each policy, with the exception that StudentLoans has some additional information but we can tackle that later.
Eventually we'll get to a point where a service operator can mark these checks as completed or not.
Screenshots

 Introduce claim checking task list

I started writing the code to introduce the ITT specialism/specialism subject etc and found myself duplicating a lot of the conditional logic we wrote in EligibilityAdminAnswersPresenter#answers.
I decided it wouldn't be too much extra work to refactor the code slightly so we could reuse the logic in these new checks and we get the added benefit of it working with student loans.
I'll see if I can get some time from a claims checker to make sure we haven't missed anything.

As per our conversation earlier, I've changed the code in the last commit to introduce a new presenter for the new claims checks.


Dynamic related models for relations

Hi,
I've come across a situation where it is desirable to dynamically return different models for the same relation. Please consider the following (schematic) example:
<?php

class A extends Model {

  public function doSomethingBasic()
  {
    // Logic specific to A, and abstract enough to apply to all derivates of A
  }
}

class B extends A {

  public function doSomethingBasic()
  {
    // Logic specific to B, which overrides the method of A
  }
}

class C extends A {

  public function doSomethingAdditionalAsWell()
  {
    // Logic specific to C
  }

  public function doSomethingBasic()
  {
    // Logic specific to C, which overrides the method of A
  }
}

class ParentModel extends Model {

  public function children()
  {
    return $this->hasOne(A::class);
  }
}

In the example above, all classes are derived from the base Model which ships with Eloquent. The ParentModel class has a relation with class A. Because classes B and C are derived from A, they inherit all properties (including database-level attributes such as the table name and primary key). In my specific situation, the table containing entries for A has certain fields which may or may not mean the related model should actually be a child class of A, such as application-level logic that A doesn't need, but B or C would.
Based on the data of each row of the table containing A entries, the Collection (or single model instance) returned from the relation call would be instances of Aor instances that are derived from A (in accordance to the Liskov Substitution).
Basically, this particular problem would be solved if I could simply hook into the part of the routine that creates an instance of the class that is passed into the Relation class, and being able to override that logic with my own. I would suggest to alter the relationship methods of Eloquent's Model class to make it possible to pass a closure (as an additional method argument) which decides what instance is finally returned. That closure would simply accept a single array of data which would represent a single row of data from the database table.
Thoughts?

Hi, I'm not sure, but (as I think it's SingleTableInheritance) and u can do:
class A extends Model
{
...
public function newFromBuilder($attributes = array())
{
switch( $id = array_get($attributes, 'id') ){
case 75:
return new \B($attributes);
        case 368:
            return new \C($attributes);

        default:
            return parent::newFromBuilder($attributes);
    }
}

...
Sry for cucumber eng ))

I'm sorry, but we don't process proposal issues. Feel free to discuss on slack and/or the forums, and possibly send a PR depending on the results of those discussions.


Not really an issue, but moreso a question.

Is there any way to adjust camera sensitivity or change any of the controls? I'm checking all the files but can't seem to find a way.

Probably not without delving into the code. From my understanding, this is designed to be a game engine rather than a complete game.


Android build error: Execution failed for task ':react-native-theoplayer:compileDebugKotlin'

"react-native": "0.68.7",
"react-native-theoplayer": "^7.6.0", (but also happens on 7.4.0)
kotlinVersion = "1.9.0"

Would love to know if anyone has any feedback or solutions! Let me know if you have any questions about a version of anything I'm using
@tvanlaerhoven

Hey @davidwinograd1 , we indeed had an unintentional API change in our native Android player SDK. It was a breaking change as it added an abstract method to some Ad classes. For that reason, if you'd upgrade to react-native-theoplayer v7.4.0 or higher, you would have to combine this with an Android player SDK 7.6.0+.
We included this dependency constraint in the SDK's gradle file:
https://github.com/THEOplayer/react-native-theoplayer/blob/develop/android/build.gradle#L116
Could it be you have pinned your Android player SDK to a lower version in the gradle.properties?
https://github.com/THEOplayer/react-native-theoplayer/blob/develop/example/android/gradle.properties#L44
If so, please upgrade here.

Got it! Yes it was on a lower SDK Android version. @tvanlaerhoven
I added this to my gradle.properties:
THEOplayer_sdk = 7.6.0
But then I get the error below...any ideas?

Fixed! Thank you!

@davidwinograd1 OK good to hear!


schedule synching between amazon instance state and jenkins worker idleness

sometimes jenkins forgets to stop an instance (ok ok, not necessarily its fault) --> run every 30 min and stop all instances whose workers are idle

Use code along the lines of (from DoubleLaunchChecker):
import hudson.triggers.SafeTimerTask;
import jenkins.util.Timer;

Timer.get.schedule(new CheckNodesTask, (random.nextInt(30) + 60) * 1000 * 60, TimeUnit.MILLISECONDS)
class CheckNodesTask extends SafeTimerTask {
  def doRun = ...
}


Nesting takes too long and nothing happens

Hello,
I make drawings on CorelDraw. I save them in SVG, open them in inkscape, ungroup everything, convert to path and save again.
Even so SVGnest does not generate a solution in a reasonable time.
Here the last SVG file I tried: https://goo.gl/86TEHx
In commercial software the nesting was conclued in less than 10 seconds.
Printscreen: https://goo.gl/qxHMMg
SVGnest is running now for more than 20 minutes and nothing happen. Windows Task Manager show me 99% CPU usage.
Printscreen: https://goo.gl/15L1eV
I'm trying with these configurations options: https://goo.gl/UVRZew

I had a similar issue with a dummy SVG where the bounding box was very large. I made it smaller and SVGnest worked.


Implement handling of Juniper BGP traps (jnxBgpM2BackwardTransition and jnxBgpM2Established from BGP4-V2-MIB-JUNIPER)

Zino 1 appears to ignore traps from the regular IETF BGP4-MIB, and only handles traps from the Juniper-specific BGP MIB BGP4-V2-MIB-JUNIPER.
Mainly two traps are handled:

jnxBgpM2BackwardTransition - indicates that a BGP peer was lost.
jnxBgpM2Established - indicates that a BGP peer came back up.

Closed by #291


When render config fail, we may not aware it.

          In the end of the day what is considered to have the exporter installed?

In my understanding reading the code is:

exist a file at /etc/systemd/system/hardware-exporter/.service and /etc/hardware-exporter-config.yaml

This is very easy to check and not CPU intensive by using Path.exists and I don't see the need at all of using self._stored and by consequence those type ignore.
The juju sdk documentation has an interesting section of uses and limitations with StoredState and the problematic solution looks like a lot with what I'm seeing in this project.
I've spotted another possible issue into the install function:
https://github.com/canonical/hardware-observer-operator/blob/86f0d7e0dd8b0339740518b27eb224fb1fcaf2b9/src/service.py#L132-L149
See that if line 137 fails, it will overwrite the result on line 143 meaning that if for some reason the redfish config file fails, but for the service doesn't, you won't notice.
To say the truth I' would refactor those functions of install and uninstall to return nothing and let to bubble in the charm code here
Originally posted by @gabrielcocenza in https://github.com/canonical/hardware-observer-operator/pull/192#discussion_r1536122008

As part of this PR to add smartctl exporter support for the charm, we are removing the usage of Stored State to check whether an exporter is installed.
In the same PR, we're also using separate variables (render_config_success, render_service_success) and so we won't face this issue in the exporter install function. Refer here.


Add fetch-depth option to checkout action

This should hopefully stop the merge conflicts when rebasing in github actions

For future context https://github.com/actions/checkout/issues/372

For future context https://github.com/actions/checkout/issues/372


Update version

v1.10.0

:tada: This PR is included in version 1.0.0 :tada:
Your semantic-release bot :package::rocket:


lzip and zstd dependencies seem gratuitous

As far as I can see, both of these are used to unpack a single package each (binutils and GMP, respectively). Both of those projects also publish .xz packages that are roughly of the same size. It would make it a bit easier to set up the build environment if you were to drop those dependencies and use the .xz packages instead.

While I agree with your analogy for lzip and xz, I have to disagree with zstd as its used by GCC to provide a compression backend for LTO instead of plain zlib.
Feel free to submit a PR, and I'll hapilly review and merge it.

I have to disagree with zstd as its used by GCC to provide a compression backend for LTO instead of plain zlib.

Okay, but for that you need the zstd development files, right? Not the command-line tool.

You're mostly correct, but not all host distributions provide split packages (e.g. Arch).


[Meta] Automate Run backward compatibility tests for the distribution (OpenSearch + plugins)

Background
Currently, each plugin has their own backward compatibility tests running in their GitHub actions. It sets up their own test cluster and test each individual subsequent tasks.
Here is our proposal for BWC/Upgrade testing for OpenSearch project.
All BWC tests include:

Rolling upgrade
Mixed Cluster search
Snapshot upgrade
Restart upgrade

In plugin level, the current bwc tests tasks are running against the test cluster framework from the core team, which set up the test cluster with designated OpenSearch min artifacts. Within the test cluster on plugin teams end, it only installs their own plugin and other required dependencies plugins.
Proposal
We would like to see CI/CD that shows each of the plugin would be tested with the latest bundle build of OpenSearch. We like to run backward compatibility tests once the distribution is ready. In another word, running the plugin BWC against the test clusters with all plugins present along with the OpenSearch min once we generate a new bundle distribution.
Currently plugin teams are using the test cluster set up from TestClustersPlugin, within this framework, plugin teams can set up the number of nodes, the version of OpenSearch min artifacts...
I think we could enhance with another test cluster set up option that we could instead install just the core but with the distribution that we generated from the CI distribution build. When we started the new test cluster
testClusters {
    "mycluster" {
        versions = ["1.3.2", "2.4.0"]
        numberOfNodes = 4
    }
}

In the example above, each of those 4 nodes are set up to be started OS version 1.3.6 with all plugins running and the second is the one set up to be the current project version. In this framework, each node will initially be downloaded with DistributionDownloadPlugin and installed with the respective bundle verison. From plugin perspectives, if they want to run BWC tests against the bundle test cluster, they could add a new argument to start the tests.
Works breakdown

Milestone 1: Update the distributionDownload framework to download bundle distribution from CI build.
Milestone 2: Onboard the updated gradle BWC tests to plugin repos/ Adapt current BWC gradle tasks to the new test clusters with bundle installed in one plugin
Milestone 3: Onboard the BWC tests to CI workflow running associated with CI build
Milestone 4: Report failures for any CI tests failures to plugins teams

Idea here is to download the bundle from CI bucket with latest index.
e.g.
v1.3.6 Bundle URL: https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/1.3.6/latest/linux/x64/tar/dist/opensearch/opensearch-1.3.6-linux-x64.tar.gz
v2.4.0 Bundle URL: https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/2.4.0/latest/linux/x64/tar/dist/opensearch/opensearch-2.4.0-linux-x64.tar.gz

The URLs used before for test cluster setup are following this pattern.
https://artifacts.opensearch.org/releases/core/opensearch/1.3.6/opensearch-min-1.3.6-linux-x64.tar.gz for released version
https://artifacts.opensearch.org/snapshots/core/opensearch/2.4.0-SNAPSHOT/opensearch-min-2.4.0-SNAPSHOT-linux-x64-latest.tar.gz for unreleased version.
The approach is to add a new URL pattern to the distributionDownloadPlugin framework.

@zelinh Can you please provide the current status? Lets fast track the on-boarding process so that we can use 2.5.0 release as dry run to close the gaps.

Backport of the PR listed above to 2.x branch is merged. I finally have the backport PR to 2.4 branch in OpenSearch repo pass the gradle checks.
Will trigger the build on 2.4.2 and 2.5 to generate the new build-tools with these changes.

Since the test cluster setup PRs were merged, I am drafting onboarding PR to component repos for the options to run BWC tests on distribution level soon. Plan to start with AD plugin first.

For the BWC version (old version) of bundle, I decided to use v1.3.2 to start with. It was the first version that we started to support our current CI bundle URL pattern which follows as this example
https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/1.3.2/latest/linux/x64/tar/dist/opensearch/opensearch-1.3.2-linux-x64.tar.gz

I have onboarded the BWC tests running on distribution level to AD, notification, sql, observability, alerting, index-management plugins.
There are also some other plugins that haven’t been onboarded with BWC tests on distribution level.
- One reason is that these repos’ currently BWC tests on plugin level are failing or unstable. We would need to work with them to fix the test structures or assertions before onboarding with new tests (e.g. cross-cluster-validation, reporting).
- Another reason is that most of our un-onboarded plugins are pretty new and don’t have enough versions/sources for running BWC tests. We would sequentially work on onboarding once they start to have BWC tests. (e.g. ml-commons, geospatial, neural-search)

@zelinh Whats pending for BWC tests? Can we add automated GitHub issue creation in this workflow as well @rishabh6788

Closing this issue.

Hey @zelinh quick question, does this updated gradle BWC tests part of every plugin repo or just the few? Also when you get a chance can you please update the README of backwards-compatibility-tests section part of the test workflow. It would be very helpful for this OpenSearch Release Process documentation and I dont need to add all the details again to the release process doc whereas I can just refer to section part of the test workflow.
Thank you

@prudhvigodithi Right now, the BWC tests on distribution bundle is only run on few plugins that were onboarded with BWC tests. Yes, I add update the README in our repo regarding of this. I have updated in the plugin repo and could have those docs added here as well.

Thanks @zelinh


⚠ Bofrost.be has degraded performance

In c7d1648, Bofrost.be (https://www.bofrost.be) experienced degraded performance:

HTTP code: 200
Response time: 1200 ms

Resolved: Bofrost.be performance has improved in 873f429.


Updates include: 1.  Allowing to set marker Z index 2.  Allowing to set marker image Anchor point 3.  Fixed issue with GoogleMapMarker and compare fields of other goodle map marker is null

Updates include:

Allowing to set marker Z index
Allowing to set marker image Anchor point
Fixed issue with GoogleMapMarker and compare fields of other goodle map marker is null

I didn't like the way that was implemented.   I'm going to fully implement the marker option and resubmit.   Please ignore


H

NOTE: Tink is moving to github.com/tink-crypto!
We are in the process of splitting Tink into multiple focused repositories.
Please create this feature request in the appropriate repository in
github.com/tink-crypto:

Tink Java
Tink Java AWS KMS extension
Tink Java Google Cloud KMS extension
Tink Java apps extension
Tink C++
Tink C++ AWS KMS extension
Tink C++ Google Cloud KMS extension
Tink Go
Tink Go AWS KMS extension
Tink Go Google Cloud KMS extension
Tink Go HashiCorp Vault KMS extension
Tink Python
Tink Tinkey

Help us help you
Tell us more about
your Tink deployment.
Is your feature request related to a problem?
If so, provide a description of the problem.
What sort of feature would you like to see?
Describe your feature request. Or, how can we help solve the problem you noted
above?
Have you considered any alternative solutions?
Provide an explanation of any alternate requests or solutions and why your
request is preferable.
Would you like to add additional context?
Provide any other context about the feature request.

At first sight this looks like an empty issue, so I am gonna go and close it. Feel free to reopen and provide more details if needed.


Proposal: please add [Iterable.all]

  /// Checks whether all elements of this iterable satisfies [test].
  bool all(bool test(E element));

Example:
// Easy to understand
var allCompleted = tasks.all((t) => t.completed);

// Hard to understand
var allCompleted = !tasks.any((t) => !t.completed);

Dart SDK Version = 2.0.0-dev.68.0

There is a method like that. It is called Iterable.every.


Add multiline strings

This PR adds multiline stings to clue. Although regular strings can span multiple lines any tab or newline characters are not preserved. This other type of multiline strings preserves those.
The ` symbol is used to denote multiline strings
print(`
hello
world
`)

When this program is run it should output
hello
world

Does it need to use its own ComplexToken? Especially for 3.0 I would need it to use SYMBOL

The problem with SYMBOL is that it  is used in places where multiline string is invalid so I wasn't sure if it was a good idea to do that

SYMBOL can be compiled anywhere, you just have to parse multilined strings in the right place

SYMBOL can be compiled anywhere, you just have to parse multilined strings in the right place

Hmm ok let me actually make a SYMBOL only implementation of this

I can see a much simpler and faster way to implement this, can I merge this and then change it?

Sure


Upgrade modernisation-platform-environments repo to AWS provider v4

User Story
As a modernisation platform engineer
I want to create documentation and provide support for teams to use the latest AWS provider version
So that they can upgrade with minimal assistance from the Modernisation Platform team
Value
Much like #1466 , we want to provide customers of the modernisation platform with the ability to make the same use of Terraform's AWS provider V4. This will ensure that they have access to the most up-to-date features supplied by the provider, as well as bug fixes and other enhancements.
Questions / Assumptions
We will supply guidance on version constraints and any code refactoring on request
We will need to help them with any terraform imports needed
Definition of done

[ ] guidance created
[ ] providers updated
[ ] all resources imported

Reference
How to write good user stories
Terraform AWS Provider Version 4 Upgrade Guide

All customers upgraded besides xhibit-portal. Given their upcoming delivery deadline we've agreed to revisit their upgrade on the week of the 28th of March.

Completed in xp-dev and xp-preprod branches. Completed in feature branch for production, merged in through https://github.com/ministryofjustice/modernisation-platform-environments/pull/600


Update README.md

Add distance parameter, as it is required in the current version

Changed to enum


Bug report:m365 spo list add fails when giving list a title containing numeric values only

Priority
(Medium) I'm annoyed but I'll live
Description
When trying to run the command m365 spo list add and providing only a numeric value in the '--title' parameter an error is returned as follows:

Error: Cannot convert a primitive value to the expected type 'Edm.String'. See the inner exception for more details.

Steps to reproduce
Run command with numeric value only in the title field, for example:
$lib = m365 spo list add --webUrl "https://$tenant.sharepoint.com/sites/$sitealias" --title "1000" --baseTemplate DocumentLibrary -o json | ConvertFrom-Json
Expected results
A document library would be created with a numeric value for the title.
Actual results

Diagnostics
Executing command spo list add with options {"options":{"webUrl":"https://tenant.sharepoint.com/sites/sitecollection","title":1004,"baseTemplate":"DocumentLibrary","output":"json","debug":true}}
Creating list in site at https://tenant.sharepoint.com/sites/sitecollection...
Existing access token eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiIsIng1dC..... still valid. Returning...
Request:
{
  "url": "https://tenant.sharepoint.com/sites/sitecollection/_api/web/lists",
  "method": "post",
  "headers": {
    "common": {
      "Accept": "application/json, text/plain, */*"
    },
    "delete": {},
    "get": {},
    "head": {},
    "post": {
      "Content-Type": "application/x-www-form-urlencoded"
    },
    "put": {
      "Content-Type": "application/x-www-form-urlencoded"
    },
    "patch": {
      "Content-Type": "application/x-www-form-urlencoded"
    },
    "user-agent": "NONISV|SharePointPnP|CLIMicrosoft365/6.7.0",
    "accept-encoding": "gzip, deflate",
    "X-ClientService-ClientTag": "M365CLI:6.7.0",
    "accept": "application/json;odata=nometadata",
    "authorization": "Bearer eyJ0eXA......"
  },
  "responseType": "json",
  "decompress": true,
  "data": {
    "Title": 1004,
    "BaseTemplate": 101
  }
}
Request error:
{
  "url": "https://tenant.sharepoint.com/sites/sitecollection/_api/web/lists",
  "status": 400,
  "statusText": "Bad Request",
  "headers": {
    "cache-control": "private, max-age=0",
    "transfer-encoding": "chunked",
    "content-type": "application/json;odata=nometadata;streaming=true;charset=utf-8",
    "expires": "Tue, 30 May 2023 11:49:54 GMT",
    "last-modified": "Wed, 14 Jun 2023 11:49:54 GMT",
    "vary": "Origin",
    "p3p": "CP=\"ALL IND DSP COR ADM CONo CUR CUSo IVAo IVDo PSA PSD TAI TELo OUR SAMo CNT COM INT NAV ONL PHY PRE PUR UNI\"",
    "x-networkstatistics": "0,2102272,38,113,486795,0,2102272",
    "x-sharepointhealthscore": "1",
    "x-sp-serverstate": "ReadOnly=0",
    "dataserviceversion": "3.0",
    "spclientservicerequestduration": "13",
    "sprequestduration": "13",
    "x-aspnet-version": "4.0.30319",
    "x-databoundary": "EU",
    "x-1dscollectorurl": "https://eu-mobile.events.data.microsoft.com/OneCollector/1.0/",
    "x-ariacollectorurl": "https://eu-mobile.events.data.microsoft.com/Collector/3.0",
    "sprequestguid": "a8bebca0-101f-6000-a96f-0ce8142c6543",
    "request-id": "a8bebca0-101f-6000-a96f-0ce8142c6543",
    "ms-cv": "oLy+qB8QAGCpbwzoFCxlQw.0",
    "strict-transport-security": "max-age=31536000",
    "x-frame-options": "SAMEORIGIN",
    "content-security-policy": "frame-ancestors 'self' teams.microsoft.com *.teams.microsoft.com *.skype.com *.teams.microsoft.us local.teams.office.com *.powerapps.com *.yammer.com *.officeapps.live.com *.office.com *.stream.azure-test.net *.microsoftstream.com *.dynamics.com *.microsoft.com onedrive.live.com *.onedrive.live.com securebroker.sharepointonline.com;",
    "x-powered-by": "ASP.NET",
    "microsoftsharepointteamservices": "<IP_ADDRESS>24",
    "x-content-type-options": "nosniff",
    "x-ms-invokeapp": "1; RequireReadOnly",
    "x-cache": "CONFIG_NOCACHE",
    "x-msedge-ref": "Ref A: 78CFF7F3DC1346678942D0D482774177 Ref B: MAN31EDGE0413 Ref C: 2023-06-14T11:49:54Z",
    "date": "Wed, 14 Jun 2023 11:49:54 GMT",
    "connection": "close"
  },
  "error": {
    "odata.error": {
      "code": "-1, Microsoft.SharePoint.Client.InvalidClientQueryException",
      "message": {
        "lang": "en-US",
        "value": "Cannot convert a primitive value to the expected type 'Edm.String'. See the inner exception for more details."
       See the inner exception for more details.

CLI for Microsoft 365 version
6.7.0
nodejs version
v14.15.0
Operating system (environment)
Windows
Shell
PowerShell
cli doctor
{
  "os": {
    "platform": "win32",
    "version": "Windows 10 Enterprise",
    "release": "10.0.22621"
  },
  "cliVersion": "6.7.0",
  "nodeVersion": "v14.15.0",
  "cliAadAppId": "31359c7f-bd7e-475c-86db-fdb8c937548e",
  "cliAadAppTenant": "common",
  "authMode": "DeviceCode",
  "cliEnvironment": "",
  "cliConfig": {},
  "roles": [],
  "scopes": [
    "AllSites.FullControl",
    "AppCatalog.ReadWrite.All",
    "AuditLog.Read.All",
    "Bookings.Read.All",
    "ChannelMember.ReadWrite.All",
    "ChannelMessage.Read.All",
    "ChannelMessage.Send",
    "ChannelSettings.ReadWrite.All",
    "Chat.Read",
    "Chat.ReadWrite",
    "Directory.AccessAsUser.All",
    "Directory.ReadWrite.All",
    "ExternalConnection.ReadWrite.All",
    "Group.ReadWrite.All",
    "IdentityProvider.ReadWrite.All",
    "Mail.Read.Shared",
    "Mail.ReadWrite",
    "Mail.Send",
    "Notes.Read.All",
    "Place.Read.All",
    "Policy.Read.All",
    "RecordsManagement.ReadWrite.All",
    "Reports.Read.All",
    "SecurityEvents.Read.All",
    "ServiceHealth.Read.All",
    "ServiceMessage.Read.All",
    "ServiceMessageViewpoint.Write",
    "Tasks.ReadWrite",
    "Team.Create",
    "TeamMember.ReadWrite.All",
    "TeamsAppInstallation.ReadWriteForUser",
    "TeamSettings.ReadWrite.All",
    "TeamsTab.ReadWrite.All",
    "TermStore.ReadWrite.All",
    "User.Invite.All",
    "User.ReadWrite.All",
    "profile",
    "openid",
    "email",
    "AllSites.FullControl",
    "AppCatalog.ReadWrite.All",
    "AuditLog.Read.All",
    "Bookings.Read.All",
    "ChannelMember.ReadWrite.All",
    "ChannelMessage.Read.All",
    "ChannelMessage.Send",
    "ChannelSettings.ReadWrite.All",
    "Chat.Read",
    "Chat.ReadWrite",
    "Directory.AccessAsUser.All",
    "Directory.ReadWrite.All",
    "ExternalConnection.ReadWrite.All",
    "Group.ReadWrite.All",
    "IdentityProvider.ReadWrite.All",
    "Mail.Read.Shared",
    "Mail.ReadWrite",
    "Mail.Send",
    "Notes.Read.All",
    "Place.Read.All",
    "Policy.Read.All",
    "RecordsManagement.ReadWrite.All",
    "Reports.Read.All",
    "SecurityEvents.Read.All",
    "ServiceHealth.Read.All",
    "ServiceMessage.Read.All",
    "ServiceMessageViewpoint.Write",
    "Tasks.ReadWrite",
    "Team.Create",
    "TeamMember.ReadWrite.All",
    "TeamsAppInstallation.ReadWriteForUser",
    "TeamSettings.ReadWrite.All",
    "TeamsTab.ReadWrite.All",
    "TermStore.ReadWrite.All",
    "User.Invite.All",
    "User.ReadWrite.All"
  ]
}

Additional Info
No response

Hi @adam13hylo, thanks for reporting this, I'm sorry you had to run into it. I can see why this is happening, it turns out the CLI interprets the value as a number before sending it to SharePoint. I think we can fix this quite easily by typing the option as a string. We'll try to ship a fix as fast as possible 👍


Java VMs?

Your repos on benchmarks are amzing! There'is so much interesting stuff!
Do you plan to include any of the available Java VMs in this benchmark? Do you find it appropriate/interesting?
Thank you

what of java vm you mean? scala?

what of java vm you mean? scala?

there're multiple Java implementations: HotSpot (most common), OpenJdk, OpenJ9, Graal... in multiple versions
Java is also highly tunable via parameters. For example, you can choose a different Garbage Collector https://blogs.oracle.com/javamagazine/understanding-the-jdks-new-superfast-garbage-collectors
this applies to Java-based runtimes like JRuby and Jython
(and TruffleRuby, but it does support GraalVM only)
Scala is also a language implement on top of the Java runtime
it might be interesting to compare different Java VMs, but I think it's out of the scope of this repository.

there're multiple Java implementations: HotSpot (most common), OpenJdk, OpenJ9, Graal... in multiple versions
Java is also highly tunable via parameters. For example, you can choose a different Garbage Collector https://blogs.oracle.com/javamagazine/understanding-the-jdks-new-superfast-garbage-collectors
this applies to Java-based runtimes like JRuby and Jython
(and TruffleRuby, but it does support GraalVM only)
Scala is also a language implement on top of the Java runtime
it might be interesting to compare different Java VMs, but I think it's out of the scope of this repository.

what of java vm you mean? scala? i not realy familiar with java world.

I am not an expert either. Basically the Java programming language is first translated into bytecode (and you get .jar files) and then compiled at runtime (that´s why I thought it would qualify here, it is JIT compilation) in a virtual machine (VM).
If you look here:
https://en.wikipedia.org/wiki/Just-in-time_compilation
you would see that Java has JIT compilation. As @ahorek said above, there are various types of Java VM and various versions. It would be very interesting to compare at least some of the major ones.
If you want, I could try and help.

what of java vm you mean? scala? i not realy familiar with java world.

I am not an expert either. Basically the Java programming language is first translated into bytecode (and you get .jar files) and then compiled at runtime (that´s why I thought it would qualify here, it is JIT compilation) in a virtual machine (VM).
If you look here:
https://en.wikipedia.org/wiki/Just-in-time_compilation
you would see that Java has JIT compilation. As @ahorek said above, there are various types of Java VM and various versions. It would be very interesting to compare at least some of the major ones.
If you want, I could try and help.

there're multiple Java implementations: HotSpot (most common), OpenJdk, OpenJ9, Graal... in multiple versions
Java is also highly tunable via parameters. For example, you can choose a different Garbage Collector https://blogs.oracle.com/javamagazine/understanding-the-jdks-new-superfast-garbage-collectors
this applies to Java-based runtimes like JRuby and Jython
(and TruffleRuby, but it does support GraalVM only)
Scala is also a language implement on top of the Java runtime
it might be interesting to compare different Java VMs, but I think it's out of the scope of this repository.

Could you please clarify why would it be out of scope in this repository?

there're multiple Java implementations: HotSpot (most common), OpenJdk, OpenJ9, Graal... in multiple versions
Java is also highly tunable via parameters. For example, you can choose a different Garbage Collector https://blogs.oracle.com/javamagazine/understanding-the-jdks-new-superfast-garbage-collectors
this applies to Java-based runtimes like JRuby and Jython
(and TruffleRuby, but it does support GraalVM only)
Scala is also a language implement on top of the Java runtime
it might be interesting to compare different Java VMs, but I think it's out of the scope of this repository.

Could you please clarify why would it be out of scope in this repository?

I think it's worth a separate project.

I think it's worth a separate project.

Given the current list of scripting/high-level dynamically-typed languages, I think Java wouldn't really fit here.
Java is not used like Ruby/Python/JS/Lua is.
That's just my opinion though.
It would probably make more sense to compare Java to Crystal or so, but in this repo C/Crystal are used as baselines.
Maybe a single version & VM of Java would make sense as a baseline?
As mentioned, there are many implementations of Java, but probably 95+% use HotSpot, so the rest is fairly niche, and most likely wouldn't make much sense for a baseline.

Given the current list of scripting/high-level dynamically-typed languages, I think Java wouldn't really fit here.
Java is not used like Ruby/Python/JS/Lua is.
That's just my opinion though.
It would probably make more sense to compare Java to Crystal or so, but in this repo C/Crystal are used as baselines.
Maybe a single version & VM of Java would make sense as a baseline?
As mentioned, there are many implementations of Java, but probably 95+% use HotSpot, so the rest is fairly niche, and most likely wouldn't make much sense for a baseline.


fix laggy first paint

when first launching the app, the scroll from page A to page B is very laggy due to the first successful paint, we should try to minimize this by trying to implement a future building process (back buffer rendering kinda strategy).
the effect goes away after flutter has cached all of the pages

ok not really possible unless we want a really laggy major repaint lol


ParameterDateFormat support

Currently, Serialization of C# DateTime objects when used as Parameters can be configured by the Parameter ParameterDateTimeFormat. However, sometimes the DateTime object should be treated as Date and not DateTime.
Therefore, I added the parameter ParameterDateFormat that defines the Format when only the date is required.
Example:
"paths": {
    "/v1.0/somePath}/{fromDate}/{to}": {
      "get": {
        "tags": [ "example"  ],
        "parameters": [
          {
            "name": "fromDate",
            "in": "path",
            "schema": {
              "type": "string",
              "format": "date"
            },
          {
            "name": "to",
            "in": "path",
            "schema": {
              "type": "string",
              "format": "date-time"
            },
          }

Remark:
Since, I'm not too familiar with NSWAG code structur, so it could be that this changes does not fit in codewise.  So could very well be if this change is not ok.
 The only thing missing to ensure the feature is working is a unit test. Will review tomorrow i hope.

yes.. unit test is missing. Ill try to find the right way to implement one. (might take me some time).

Added a unit test.. not sure if it is in the right place though.

Great thank you.
Another note, is there a possibility to have a reference api that is to be used for integration tests?
It would then be possible to use the various generated clients to call the api and one could test the outcome (api might write to a file for example).
Not sure if its a good or realistic idea but something like this would have quite a big test coverage. Inital setup is probably a pain though.

It would be great to have a full integration test, a reference api, then generate clients and then calling the api with the clients - all automatic in a test without publishing a new version.
At the moment we only check the output of the generators but not whether they actually work
But this is quite some work

In the typeScript generator, there is still no differences between dates and date-times
I've started to take a look at it, but it's more complicated as we can choose the datetime type.
It would require to use some logic already present in NJsonSchema to render the correct typeScript.

is this working already ?
i still don't see the new date time format applied when using visual studio add service reference, and passing this as parameters, am i doing something wrong?
does anyone have a way to verify in vs2019/windows ?
<ItemGroup>
    <!-- https://github.com/RicoSuter/NSwag/wiki/CSharpClientGeneratorSettings -->
    <OpenApiReference Include="OpenApi\MySpecWithDateTimes.json">
      <Options>/GenerateClientClasses:false /GenerateClientInterfaces:false /AllowReferencesWithProperties:true /ParameterDateTimeFormat:"yyyy-MM-ddTHH:mm:ss"</Options>
    </OpenApiReference>
  </ItemGroup>

generated properties are still in DateTimeOffset (offset is giving us trouble as it's not in date-time swagger default), and, don't see any format attribute applied to them

ParameterDateTimeFormat and ParameterDateFormat refer to the dates that are in the path and thus not serialized by newtonsoft (or by some other json serializer in the latest core version). If I'm not mistaken, the format of dates that are in the body can be controlled by the serializer settings.
PS: jkone27: you can contact me directly. I dont consider myself an expert on NSWAG but still might be able to help in this matter.


final-fix-#4603

Fix according to @keturn suggestion. Sorry I had to made a separate PR because I accidentally deleted the previous branch.

just trying to understand why we need to restore where the window was placed on the screen.
do you still need these PRs:

https://github.com/MovingBlocks/Terasology/pull/4603
https://github.com/MovingBlocks/Terasology/pull/4601

This is a change to the shutdown method, so I think it replaces #4603 but not the one that changes the initialise method.
The code looks good to me. We'll need someone else to test it because I wasn't able to reproduce this under Ubuntu.

just trying to understand why we need to restore where the window was placed on the screen. what does GLFW do by default if you don't specify where the window is placed? is that up to the windows manager to drive that or is it by default placed at some fixed value like (0,0).
do you still need these PRs:

#4603
#4601

Yes if we don't restore where the window was placed, it will always start with default values in #4160.
No those PRs won't be ne necessary anymore. Shall I remove those?

This is a change to the shutdown method, so I think it replaces #4603 but not the one that changes the initialise method.
The code looks good to me. We'll need someone else to test it because I wasn't able to reproduce this under Ubuntu.

Okay :)


feat: Add Preact tests

Please describe the changes this PR makes and why it should be merged:
Status

[] Code changes have been tested against prettier, or there are no code changes
[] I know how to update typings and have done so, or typings don't need updating

Semantic versioning classification:

[x] This PR changes the codebase

[ ] This PR includes breaking changes (methods removed or renamed, parameters moved or removed)
[ ] This PR changes the internal workings with no modifications to the external API (bug fixes, performance improvements)

[ ] This PR only includes non-code changes, like changes to documentation, README, etc.

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 37.49%

Totals

Change from base Build<PHONE_NUMBER>:
0.0%

Covered Lines:
822

Relevant Lines:
2306

💛  - Coveralls

You can check now @aidenybai. Let me know if there is anything else to fix.


keg_relocate: fix all text files being marked as changed

This was wildly inflating our image indexes.
Also add an abort before upload if the image index is >= 4MB (assuming 1024 * 1024 bytes = MB here).

The check is working:
==> Uploading asyncapi--0.51.3.arm64_monterey.bottle.tar.gz
Error: Image index too large!

https://github.com/Homebrew/homebrew-core/actions/runs/5689779966/job/15421922953


Make recordset horizontal scrollbar sticky

We would like to make the horizontal scrollbar that we're displaying above the table to be sticky.
If we only want this on the recordset page, we can follow what we did for the recordedit page. But if we want this to work on all the recordset table instances, we probably need to develop a new solution.
P.S. On the AngularJS version of recordset page we tried making the whole table header sticky, but given the issues we faced, we didn't merge the changes. The approaches we tried and could try with React are documented in #2101.

Common approaches to make a scrollbar sticky and the issues associated with them can be found here - https://github.com/informatics-isi-edu/chaise/issues/2101
For a generic solution, a combined approach including the usage of IntersectionObserver and the solutions mentioned in https://github.com/informatics-isi-edu/chaise/issues/2101  needs to be taken.
We wrap the table in two elements to identify the start and end of the table and observe them to determine if the table is fully / partially visible on the screen.
This information can be used to set the position of the scrollbar using the position: "sticky" and top: "__px" css properties.
Known issues -

The updates based on the IntersectionObserver events may not always reflect immediately.
Making the header sticky using the same properties causes the table columns to shrink and be misaligned with the header.

Places this should be implemented:

Recordset table in recordset app

can be tested here but more "rows" would have to be added before it can really be tested
https://dev.derivacloud.org/chaise/recordset/#84667/defaults:defaults-table@sort(RID)

Related tables in record app

https://dev.derivacloud.org/chaise/record/#84629/product-add:file/RID=23A

recordset popup modal

go to the link below, scroll to accommodation_image, and click "Link Records"
https://dev.derivacloud.org/chaise/record/#84629/product-add:file/RID=23A

This can be compared to the existing implementation in recordedit app:

clone 6 more forms to get the scroll bar to start showing
https://dev.derivacloud.org/chaise/recordedit/#84667/defaults:defaults-table


Migrate repositories from senzing to senzing-garage

Tasks

In repository:

[ ] In .github/workflows/, point to GitHub actions in senzing-factory`
[ ] In README.md articulate that this is a "garage"
[ ] Update the .github/CODEOWNERS/ file
[ ] Delete PULL_REQUEST_TEMPLATE.mdfile
[ ] Delete .github/ISSUE_TEMPLATE folder
[ ] Change reference URLs from https://github.com/Senzing to https://github.com/senzing-garage

[ ] GitHub > [repository] > Settings

[ ]  General

[ ] Pull Requests

[ ] Uncheck Allow merge commits
[ ] Check Automatically delete head branch

[ ] Collaborators and teams > Manage Access

[ ] Set Admin permission on "Senzing Garage Admin"
[ ] Set Write permissions on remaining teams

[ ] Pages > Build and deployment > Branch

[ ] Set to main and /docs

[ ] Code security and analysis

[ ] Access to Alerts

[ ] Add senzing-garage/senzing-engineering

Current proposal:

Repository
Senzing
senzing-garage
senzing-factory
Description
Notes

awesome
:white_check_mark:

Curated list of awesome software and resources for Senzing, The First Real-Time AI for Entity Resolution.
Needs to be massively updated

aws-cloudformation-database-cluster

:white_check_mark:

AWS Cloudformation with VPC and Senzing database cluster

aws-cloudformation-dev-rest

:white_check_mark:

AWS Cloudformation for developers using Senzing HTTP REST API

aws-cloudformation-ecs-senzing-stack-basic

:white_check_mark:

AWS Cloudformation demonstrating a Senzing stack that can be used with aws-cloudformation-database-cluster.

aws-cloudformation-ecs

:white_check_mark:

:no_entry: [DEPRECATED] Contains deprecated AWS Cloudformation templates

aws-cloudformation-performance-testing

:white_check_mark:

Senzing AWS performance testing project.

aws-cloudformation-webapp-demo

:white_check_mark:

Senzing WebApp demo for AWS.

aws-lambda-cognito-authorizer

:white_check_mark:

Lambda function for authorizing via AWS Cognito

aws-lambda-self-signed-certificate

:white_check_mark:

Creates a self-signed SSL certificate for AWS demonstrations.

aws-marketplace-evaluation
:question:

AWS Marketplace offering.
Not sure about this.

azure-template-aks-poc-simple

:white_check_mark:

An Azure ARM template for bringing up Senzing on Kubernetes (AKS)

bizname-research

:white_check_mark:

[Research] Determine language/country origin of business name
Current research

build-resources

:white_check_mark:
Collection of utilities to aid with build process

charts

:white_check_mark:

Helm Charts for use with Kubernetes, OpenShift, and Rancher.

check-self

:white_check_mark:

Checks environment

code-snippets
:white_check_mark:

Task based code snippet examples.

community-map

:white_check_mark:

:construction: [Under construction] A visual representation of the Senzing Community.

community-roadmap

:white_check_mark:

A peek into the future of the Senzing Community.

compressedfile
:white_check_mark:

Distributed with Senzing API package

configurator

:white_check_mark:

Web service for configuring Senzing.

connector-neo4j
:white_check_mark:

Transform Senzing data into Neo4j graph database.

data-encryption-aes256cbc-sample
:white_check_mark:

This is a sample encryption plugin, for use with the G2 engine to encrypt sensitive data in the data repository.

data-mart-replicator
:white_check_mark:

:construction: Data mart

demo-entity-search

:white_check_mark:

A CLI tool for demonstrating entity searching

discussions
:white_check_mark:

Repository that supports GitHub Discussions.

docker-adminer

:white_check_mark:

Dockerfile wrapping adminer, a database viewer.

docker-apt

:white_check_mark:

Dockerfile wrapping apt-get, a package manager.

docker-aptdownloader

:white_check_mark:

Dockerfile wrapping apt-get install --download-only.

docker-base-image-debian

:white_check_mark:

A base docker image for Senzing processes built on Debian.

docker-compose-air-gapper

:white_check_mark:

Create a TGZ bundle for air-gapped environments based on docker-compose.yaml

docker-compose-demo

:white_check_mark:

Demonstrate Senzing stack using docker-compose.

docker-db2-driver-installer

:white_check_mark:

Install DB2 client drivers on mounted volumes.

docker-hello-world

:white_check_mark:

Dockerfile for testing docker formations.

docker-ibm-db2

:white_check_mark:

Dockerfile wrapping ibmcom/db2 docker image.

docker-init-container

:white_check_mark:

Dockerfile used to initialize Senzing artifacts.

docker-installer

:white_check_mark:

Dockerfile use to install Senzing in a non-root container.

docker-mysql-init

:white_check_mark:

Dockerfile for initializing mysql database with a one-time command.

docker-python-demo

:white_check_mark:

Dockerfile demonstrating simple Flask app using Senzing.

docker-senzing-base

:white_check_mark:

Dockerfile used in FROM statements.

docker-senzing-console

:white_check_mark:

Docker-based console enabled for interacting with Senzing programs.

docker-sshd

:white_check_mark:

Dockerfile wrapping sshd.

docker-terraform-aws

:white_check_mark:

:construction: [Under construction]

docker-test-layers

:white_check_mark:

A Repository for testing contents of Docker layers

docker-test

:white_check_mark:

Used in Spikes. Not for production.

docker-web-app-demo
:white_check_mark:

Dockerfile combining Senzing API server and Senzing Entity WebApp.

docker-wrap-image-with-db2

:white_check_mark:

Wrap a Docker image with enablement for Db2 database

docker-wrap-image-with-mssql

:white_check_mark:

Wrap a Docker image with enablement for MS SQL database

docker-wrap-image-with-mysql

:white_check_mark:

Wrap a Docker image with enablement for MySQL database

docker-wrap-image-with-senzing-apt

:white_check_mark:

Wrap an existing docker image with the Senzing package.

docker-wrap-image-with-senzing-data

:white_check_mark:

Add /opt/senzing/data to a base image.

docker-xterm

:white_check_mark:

Web-based X-terminal.

docker-yum

:white_check_mark:

Dockerfile wrapping yum, a package manager.

dockerhub-util

:white_check_mark:
Utilities for working with hub.docker.com registry

dumpstack
:white_check_mark:

Distributed with Senzing API package

elasticsearch
:white_check_mark:

Using G2 engine with ElasticSearch indexing engine.

entity-search-web-app-console

:white_check_mark:

A side-car for the Entity Search Web App

entity-search-web-app
:white_check_mark:

A lightweight http server providing a web UI for entity search through the senzing api server.

evaluate

:white_check_mark:

Information for evaluating Senzing at http://senzing.com/evaluate

explain

:white_check_mark:

A CLI tool for explaining errors

file-loader

:white_check_mark:

Utility to load mapped JSON data and process redo records after loading.

g2-python
:white_check_mark:

Tools found at g2/python in the SenzingAPI package.
May no longer be needed.

g2-sdk-go-base
:white_check_mark:

An SDK for calling the Senzing C library locally.

g2-sdk-go-grpc
:white_check_mark:

An SDK for calling the Senzing C library via gRPC network requests.

g2-sdk-go-mock
:white_check_mark:

A mock object for the https://github.com/Senzing/g2-sdk-go interface.

g2-sdk-go
:white_check_mark:

Senzing's Go Software Development Kit (SDK)

g2-sdk-java
:white_check_mark:

Java SDK hosted on MvnRepository.

g2-sdk-json-type-definition

:white_check_mark:

[Spike] Codify JSON structure

g2-sdk-proto
:white_check_mark:

Proto files for the G2 Software Development Kit

g2-sdk-python-grpc
:white_check_mark:

An SDK for calling the Senzing C library via gRPC network requests.

g2-sdk-python-next
:white_check_mark:

[Development] Forthcoming major revision of Senzing Python SDK

g2-sdk-python
:white_check_mark:

Senzing Python SDK V3

g2anon
:white_check_mark:

Python SDK for G2Anonymizer

g2audit
:white_check_mark:

Distributed with Senzing API package

g2command
:white_check_mark:

Distributed with Senzing API package

g2configtables
:white_check_mark:

Distributed with Senzing API package

g2configtool
:white_check_mark:

Distributed with Senzing API package

g2createinstance
:white_check_mark:

Create a per-user instance of Senzing in a new folder with the specified name.

g2createproject
:white_check_mark:

Distributed with Senzing API package

g2database
:white_check_mark:

Distributed with Senzing API package

g2explorer
:white_check_mark:

Distributed with Senzing API package

g2export
:white_check_mark:

Distributed with Senzing API package

g2iniparams
:white_check_mark:

Distributed with Senzing API package

g2loader
:white_check_mark:

Distributed with Senzing API package

g2module
:white_check_mark:

Python SDK for G2

g2paths
:white_check_mark:

Distributed with Senzing API package

g2project
:white_check_mark:

Distributed with Senzing API package

g2s3
:white_check_mark:

Distributed with Senzing API package

g2search
:white_check_mark:

Reads formatted search records, calls Senzing, and outputs for analysis

g2setupconfig
:white_check_mark:

Distributed with Senzing API package

g2snapshot
:white_check_mark:

Distributed with Senzing API package

g2updateproject
:white_check_mark:

Distributed with Senzing API package

git-action-slack-notification

:white_check_mark:
GitHub Action for notifying Slack

github-action-add-issue-to-project

:white_check_mark:
A GitHub Action for adding an issue to projects.

github-action-docker-buildx-build

:white_check_mark:
GitHub Action for using Docker buildx command

github-action-identify-customer

:white_check_mark:
A GitAction for determining if a customer submitted the issue.

github-action-install-senzing-api

:white_check_mark:
GitHub Action for installing Senzing API

github-action-make-go-github-file

:white_check_mark:
Make a version.go file

github-action-make-go-tag

:white_check_mark:
From the Semantic Version, create a v0.0.0 tag for the Go module version

github-util

:white_check_mark:
Works with GitHub metadata.

go-cmdhelping

:white_check_mark:

Helpers for using cobra and viper

go-common

:white_check_mark:

Reusable, small components.

go-databasing

:white_check_mark:

Database routines

go-grpcing

:white_check_mark:

Go package of gRPC helpers

go-logging

:white_check_mark:

A reusable logging framework.

go-messaging

:white_check_mark:

Message creation

go-observing

:white_check_mark:

Observer software design pattern

go-queueing

:white_check_mark:

Queue routines

go-rest-api-service

:white_check_mark:

[in-progress] Go package implementation of API server

go-sdk-abstract-factory

:white_check_mark:

An abstract factory for objects of the https://github.com/Senzing/g2-sdk-go interface.

governor-postgresql-transaction-id
:white_check_mark:

Governor plugin for PostgreSQL transaction IDs.

ibm-openshift-guide

:white_check_mark:

Demonstrate Senzing stack using OpenShift and IBM Db2.

images
:white_check_mark:

Images used across repositories.

init-database

:white_check_mark:

Initialize database with Senzing schema and configuration

init-mssql

:white_check_mark:

Initialize an MS SQL database with Senzing schema and configuration

init-mysql

:white_check_mark:

Initialize a MySQL database with Senzing schema and configuration

init-postgresql

:white_check_mark:

Initialize a PostgreSQL database with Senzing schema and configuration

keystore-generator

:white_check_mark:

Generates SSL keys

knowledge-base

:white_check_mark:

HOWTOs, tasks, explanations, and more knowledge.

kubernetes-demo

:white_check_mark:

Demonstrate Senzing stack using Kubernetes.

libpostal-data
:white_check_mark:

Information about libpostal work done by Senzing.

libpostal
:white_check_mark:

A C library for parsing/normalizing street addresses around the world. Powered by statistical NLP and open geo data.

load

:white_check_mark:

A CLI tool for loading records

mapper-base
:white_check_mark:

Base functions used to map a variety of formats to a Senzing-acceptable format.

mapper-csv
:white_check_mark:

Exemplar artifacts (files) that can be used in other Senzing repositories.

mapper-dnb
:white_check_mark:

Map DNB format to Senzing format.

mapper-dowjones
:white_check_mark:

Map Dow Jones Watchlist format to Senzing format.

mapper-icij
:white_check_mark:

Map ICIJ format to Senzing format.

mapper-leie
:white_check_mark:

Map US HHS LEIE to Senzing format.

mapper-nomino
:white_check_mark:

Map Nomino format to Senzing format.

mapper-npi
:white_check_mark:

Map NPPES NPI Registry to Senzing format.

mapper-ofac
:white_check_mark:

Map OFAC to Senzing format.

mapper-open-ownership
:white_check_mark:

Map Open Ownership Register into Senzing format

mapper-opensanctions
:white_check_mark:

Map Open Sanctions into Senzing format.

mapper-safegraph
:white_check_mark:

Map SafeGraph format to Senzing format.

mapper-sayari-spark
:white_check_mark:

Map Sayari's global corporate data into Senzing format using Spark.

mapper-sayari
:white_check_mark:

Map Sayari's global corporate data into Senzing format.

move

:white_check_mark:

A CLI tool for moving data

mssql-performance
:white_check_mark:

Tips for working with Senzing on MS SQL

observe

:white_check_mark:

A CLI tool for observing

openshift-demo

:white_check_mark:

Demonstrate Senzing stack using minishift.

performance-general
:white_check_mark:

Tips about improving the performance of Senzing

postgresql-client

:white_check_mark:

A psql client that waits for the database to be ready before uploading SQL file.

postgresql-performance
:white_check_mark:

Tweaks to PostgreSQL and the Senzing DDL

rabbitmq-performance
:white_check_mark:

Tweeks to RabbitMQ

redoer

:white_check_mark:

Perform Senzing Redo operations.

resolver

:white_check_mark:

Ephemeral Senzing entity-resolution.

rest-api-client-java
:white_check_mark:

:construction: [Under construction] - Client built from OpenAPI specification.

rest-api-client-ng
:white_check_mark:

Angular TypeScript interfaces, and classes for interacting with the senzing-api-server.

risk-score-calculator
:white_check_mark:

Risk scorer.

sdk-components-ng
:white_check_mark:

A collection of UI components to interface with the Senzing Rest API server.

senzing-api-server
:white_check_mark:

Server of Senzing REST API.

senzing-commons-java
:white_check_mark:

Java classes used in multiple Senzing projects

senzing-data-encryption-specification
:white_check_mark:

This is the interface for creating encryption plugins, to work with the G2 engine to encrypt sensitive data in the data repository.

senzing-entity-specification
:white_check_mark:

Complete Senzing Entity Specification and Attribute Dictionary.

senzing-listener-example
:white_check_mark:

Example use of Senzing listener framework

senzing-listener
:white_check_mark:

Listener framework.

senzing-poc-server
:white_check_mark:

Senzing API Server plus management APIs.

senzing-rest-api-specification
:white_check_mark:

OpenAPI specification of Senzing REST API.

senzing-tools

:white_check_mark:

An aggregation of Senzing tools

senzing-tutorial-in-python

:white_check_mark:

Work in progress

senzing-with-chatgpt-notebook

:white_check_mark:

[Spike] ChatGPT and Jupyter Notebook integration

senzing-with-chatgpt-plugin

:white_check_mark:

[Spike] ChatGPT integration

senzing.github.io
:white_check_mark:

Organization site at http://hub.senzing.com

senzingapi-runtime
:white_check_mark:

Dockerfile with senzingapi-runtime binaries baked-in

senzingapi-tools
:white_check_mark:

Base image using senzingapi-tools DEB file

senzinggo
:white_check_mark:

Quickly and easily start the Senzing REST API server, demo web app & Swagger in containers.

serve-chat

:white_check_mark:

Spike for HTTP API for chat service

serve-grpc

:white_check_mark:

A gRPC server of the https://github.com/Senzing/g2-sdk-proto protocol

serve-http

:white_check_mark:

A CLI tool for serving HTTP

stream-loader

:white_check_mark:

Python tool for loading Senzing Engine from RabbitMQ, Kafka, or AWS SQS.

stream-logger

:white_check_mark:

A utility for dumping the contents of a stream to a log.

stream-producer

:white_check_mark:

Produce a stream from different input formats.

subway-map

:white_check_mark:

http://hub.senzing.com/subway-map site

support

:white_check_mark:

http://hub.senzing.com/support site

template-docker

:white_check_mark:
Best practices for creating a "docker-*" repository.

template-go

:white_check_mark:
Best practices for creating a "go-" repository.

template-python

:white_check_mark:
Best practices for creating a "python-*" repository.

template-repository

:white_check_mark:
Best practices for creating repository.

test-ground

:white_check_mark:

An area for testing ideas.  May be wiped out at any time.

truth-sets
:white_check_mark:

Distributed with Senzing API package

validate

:white_check_mark:

A CLI tool for validating data

video

:white_check_mark:

Videos available at http://senzing.com/video

Next round
GitHubMoves_v2.docx


Risk Index probably incorrect

This here is simply adding two means:
https://github.com/jxx123/simglucose/blob/c51e7f2de9918bed25b75174b8ddb21c4d960923/simglucose/analysis/risk.py#L16
However, adding up two means (or taking the mean) of sub-sequences that are different in length does not equal the mean of the whole sequence:

Instead we have to weigh the each individual mean by the length of sequence it represents. Like this:
weight_rl = len(rl) / horizon
weight_rh = len(rh) / horizon
RI = weight_rl*LBGI + weight_rh*HBGI

While risk index is not explicitly defined in any paper I can find (neither here nor here) - only the low and high risk indicies are defined:

it is not unreasonable to assume that it should be defined such that risk function applied to the whole trajectory would result in the same mean as the combination of low and high risk indicies, e.g.:

The nice thing about weighing the LBGI and HBGI is that the final risk then is still bounded between 0 and 100.

Thanks for raising this issue.
The simple add-up comes from the original UVa/Padova Simulator, but I agree that your proposed weighted average makes more sense, and it will bring the final risk index into the same scale of LBGI and HBGI. If you are interested, feel free to submit a PR to the repo, otherwise, I will do that.
FYI, there is a related risk index computation error reported here https://github.com/jxx123/simglucose/pull/62, which is due to the numpy version compatibility, and there is already a PR to fix this computation error.

With this fix, I don't really see the reason for 3 lines. Because Risk Index always overlaps either the High risk or Low risk line.

This is a follow up to my previous solution. It still suffers from a problem: the risk values are not capped to [0, 100] as they should be. This stems from the assumption (in the paper) that the bg values should be within [20, 600] mg/dL. Outside of that range we cannot depend on natural log to behave.
Here is the updated implementation - this should directly replace the contents of risk.py:
import numpy as np

def risk_index(BG, horizon):
    # BG is in mg/dL
    BG_to_compute = BG[-horizon:]
    risks =[risk(r) for r in BG_to_compute]
    LBGI = np.mean([r[0] for r in risks])
    HBGI = np.mean([r[1] for r in risks])
    RI = np.mean([r[2] for r in risks])

    return (LBGI, HBGI, RI)

def risk(BG):
    MIN_BG = 20.0
    MAX_BG = 600.0
    if BG <= MIN_BG:
        return (100.0, 0.0, 100.0)
    if BG >= MAX_BG:
        return (0.0, 100.0, 100.0)

    U = 1.509 * (np.log(BG)**1.084 - 5.381)

    ri = 10 * U**2

    rl, rh = 0.0, 0.0
    if U <= 0:
        rl = ri
    if U >= 0:
        rh = ri
    return (rl, rh, ri)

Opened a PR with the proposed changes. Closed the old PR.


⚠ Bofrost.lu has degraded performance

In ed9e34c, Bofrost.lu (https://www.bofrost.lu) experienced degraded performance:

HTTP code: 200
Response time: 2378 ms

Resolved: Bofrost.lu performance has improved in b50a215.


Can I make a restriction like credentials of API & Services in Google Cloud Platform?

I want to make my own API server from docker, but the user who can use this API server should be restricted by some conditions. For example, by specific URL referrer, by specific Android App and by specific iOS App like image below.

Hey, I think GCP Load Balancer is a way to go in your case https://cloud.google.com/load-balancing/docs/https

@lfx Thank you for your reply. But can I filter the specific iOS and Android App by GCP Load Banlancer?

@JeongJun-Lee currently you're looking for integration with Google Cloud Load Balancer or Cloud Endpoints or some API gateway product integration with Cloud Run, which does not exist. But it's on the roadmap, stay tuned!


Get web service url and key programmatically

I've been following the portal method of retrieving a machine learning web service url and key through the portal, according to https://docs.microsoft.com/en-us/azure/machine-learning/studio/consume-web-services. However, I want to know if there was a way to do this programmatically like HTTTP requests or Powershell commands?

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: e9a91d22-bd16-a250-5e81-26aca73e970b
Version Independent ID: badfb9d5-8762-830c-7601-96473d941a84
Content: Consume web service - Azure Machine Learning Studio
Content Source: articles/machine-learning/studio/consume-web-services.md
Service: machine-learning
Sub-service: studio
GitHub Login: @xiaoharper
Microsoft Alias: amlstudiodocs

@gearsmotion789 Thank you for your feedback! We will review and provide an update as appropriate.

@gearsmotion789 Here are the reference commands to get the keys of the web service.
An example is mentioned below.

Future releases of powershell might change this but it would be documented in the release notes.
We will now proceed to close this thread. If there are further questions regarding this matter, please tag @RohitMungi-MSFT in your reply. We will gladly continue the discussion and we will reopen the issue.

@RohitMungi-MSFT Should I open a new thread for this question?

@RohitMungi-MSFT Should I open a new thread for this follow-up question?


16.04 build errors

Hi all , I cloned rviz_camera_stream and started catkin_make but it gives the following errors, please advise

What version of Ubuntu and ros are you using?

ubuntu 16.04 LTS, Ros1 kinetic
On Tue, Feb 8, 2022 at 8:35 PM Lucas Walter @.***>
wrote:

What version of Ubuntu and ros are you using?
—
Reply to this email directly, view it on GitHub
https://github.com/lucasw/rviz_camera_stream/issues/42#issuecomment-1032741751,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AI7KPRUSEKAGGBQGWL5MGL3U2EZ4LANCNFSM5N2RGSWA
.
Triage notifications on the go with GitHub Mobile for iOS
https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675
or Android
https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.
You are receiving this because you authored the thread.Message ID:
@.***>

Try adding set(CMAKE_CXX_FLAGS "-std=c++0x")  ## Enable C++11 to the CMakeLists.txt.


fix inlining for good

Okay to switch that define from inline to always_inline.
However moving the functions to inlined.cpp doesn't actually do anything. When inlining is enabled the translation unit looks like this:
#include "runtime.cpp"
#include "shards/inlined.cpp"
#include "shards/core.cpp"
#include "shards/math.cpp"

So it'll already contain all those functions in the same translation unit


Hide table-row if item is sold out

By Checkbox-Control or similar. Hidden by default could be nice.

done. i close

done. i close


Fix redirect for k8s.io/docs URL

@krousey @thockin @bgrant0607

LGTM

Verified that it works on kubernetes.io


CMD.exe will run if opened as an administrator

The command prompt can be opened if run as an administrator by a user with the appropriate permissions. This is contrary to the documentation that says CMD cannot be run.

To be clear, this is referring to Windows 11 SE and the Administrative Tools section

@BagginsLongshot Can you please tell us to which article(s) your question applies? This will help us route this issue to the correct writer/team. Thanks

@BagginsLongshot We appreciate any feedback that improves the content in the Microsoft docs but we haven't had any response from you so we are going to close this issue.
Please feel free to re-open this issue if you see an improvement is required in the specific area of this documentation. We will certainly update this documentation with that valuable information. Thanks again, for taking out some time to open the issue. Appreciate and encourage you to do the same in future also.


Vaex pipeline example

Objectives:

Demonstrate reading / writing different formats and file locations
Showcase vaex capabilties & performance, with out of memory size dataset
(Showcase caching, (i.e not rerunning Task 1 if the hdf5 already exists on disk))

Try it out
$ pip install cowait
$ docker pull cowait/task

Make sure Docker is running, then:
$ cd examples/10-vaex_pipeline
$ cowait build
$ cowait run vaex_pipeline --input size xsmall

Tasks
CsvToHdf5:  Read csv from s3 and save to hdf5 on disk
Preprocess: Remove outliers and set types
TrainTestSplit: Shuffle and split into train / test
TrainModel: Task for model training, spawns multiple Fit tasks for hyper-param search
Fit: Fits the model given an alpha parameter
TestModel: Tests the winning model (training result) with the test-set
Data
NYC Taxi dataset, January 2019. Comes in 4 sizes.

size
path
rows
file size

xsmall
cowait/yellow_2019-xsmall.csv
10 000
910 KB

small
cowait/yellow_2019-small.csv
100 000
8.9 MB

medium
cowait/yellow_2019-medium.csv
1 000 000
88.9 MB

large
cowait/yellow_2019-large.csv
7 667 792
690.5 MB

TODO: Add a few more datasets (~10gb, ~100gb).
Model
Simple naive bayes classifier. Given a pickup location, the weekday and the current hour, predict the end destination of the ride.
Thoughts

I implemented minio storage and serialized FileLike input/outputs between tasks (see #161). The example on the other branch uses the new storage solution, that we think we're going to scrape.
I feel that this example is better off running on one machine (still with multiple containers), just using local disk. The minio example probably finds a better usecase for dask. This breaks kubernetes support, but perhaps that's the point - Vaex is best run on a single machine.

@johanhenriksson Try this out, --input medium takes a few minutes to run and is a good candidate for testing stuff out.
I'm getting this ocassionally. Not sure why:
164.7 fit        ! ERROR: Caught exception in fit-wnqtjfdl:
                   Traceback (most recent call last):
                     File "/opt/conda/lib/python3.7/asyncio/runners.py", line 43, in run
                       return loop.run_until_complete(main)
                     File "/opt/conda/lib/python3.7/asyncio/base_events.py", line 566, in run_until_complete
                       self.run_forever()
                     File "/opt/conda/lib/python3.7/asyncio/base_events.py", line 534, in run_forever
                       self._run_once()
                     File "/opt/conda/lib/python3.7/asyncio/base_events.py", line 1756, in _run_once
                       handle = self._ready.popleft()
                   IndexError: pop from an empty deque

                   During handling of the above exception, another exception occurred:

                   Traceback (most recent call last):
                     File "/var/cowait/cowait/worker/executor.py", line 81, in execute
                       await node.parent.send_done(result, result_type.describe())
                     File "/var/cowait/cowait/worker/parent_client.py", line 75, in send_done
                       await self.msg(TASK_RETURN, result=result, result_type=result_type)
                     File "/var/cowait/cowait/worker/parent_client.py", line 42, in msg
                       **msg,
                     File "/var/cowait/cowait/worker/parent_client.py", line 47, in send
                       await self.io.create_task(super().send(msg))
                     File "/opt/conda/lib/python3.7/asyncio/futures.py", line 260, in __await__
                       yield self  # This tells Task to wait for completion.
                   concurrent.futures._base.CancelledError
164.9 train        ERROR:Thread-1:aiohttp.server:Error handling request
                   Traceback (most recent call last):
                     File "/opt/conda/lib/python3.7/site-packages/aiohttp/web_protocol.py", line 418, in start
                       resp = await task
                     File "/opt/conda/lib/python3.7/site-packages/aiohttp/web_app.py", line 458, in _handle
                       resp = await handler(request)
                     File "/opt/conda/lib/python3.7/site-packages/aiohttp/web_middlewares.py", line 119, in impl
                       return await handler(request)
                     File "/var/cowait/cowait/network/auth_middleware.py", line 52, in middleware
                       return await handler(request)
                     File "/opt/conda/lib/python3.7/site-packages/aiohttp_middlewares/cors.py", line 234, in middleware
                       response = await handler(request)
                     File "/var/cowait/cowait/network/server.py", line 30, in handle_client
                       await self.emit(**event, conn=conn)
                     File "/var/cowait/cowait/utils/emitter.py", line 29, in emit
                       await callback(**kwargs)
                     File "/var/cowait/cowait/tasks/components/task_manager.py", line 61, in on_child_fail
                       task.set_error(error)
                     File "/var/cowait/cowait/tasks/remote_task.py", line 45, in set_error
                       self.set_status(FAIL)
                     File "/var/cowait/cowait/tasks/remote_task.py", line 36, in set_status
                       raise RuntimeError('Cant fail a completed task')
                   RuntimeError: Cant fail a completed task

I'm running OOM when running with --input size medium. This seems to happen after vaex_model.fit (it incrementally trains, but transform likely blows up memory. Giving docker 8GB on my machine (split between 3 Fit containers).

There's is a current cowait bug that sometimes gives you an error with a lower level asyncio CancelledError. This happens during websocket communications between tasks and seems to happen randomly. TBD.

This happens during websocket communications

which library are you using for this?

Cowait uses aiohttp for websocket communication.
Might be worth noting that the async cancellation bug has not appeared anywhere outside of this example (yet)

It might be interplay between vaex (using nest_asyncio) and httpio. Do you have an error msg/stack trace for me? Or do you know a way to reproduce it?

Indeed. So after some investigation, I've only managed to reproduce the problem with the Fit task.
The error only arises after the initial result has been returned, i.e there seems to be some other event after the task completes, that gives us the error. Since the upstream tasks stops caring about subtasks after they've sent  task/return, the error that comes afterwards doesn't impact execution, i.e we can complete the entire run.

{"id": "fit-tysvcgsw", "type": "task/status", "status": "fail"}
{"id": "fit-tysvcgsw", "type": "task/fail", "error": "Traceback "

Where task/fail occurs after task/return.
The stack trace in its entirety:
! ERROR: Caught exception in fit-tysvcgsw:
                   Traceback (most recent call last):
                     File "/opt/conda/lib/python3.7/asyncio/runners.py", line 43, in run
                       return loop.run_until_complete(main)
                     File "/opt/conda/lib/python3.7/asyncio/base_events.py", line 566, in run_until_complete
                       self.run_forever()
                     File "/opt/conda/lib/python3.7/asyncio/base_events.py", line 534, in run_forever
                       self._run_once()
                     File "/opt/conda/lib/python3.7/asyncio/base_events.py", line 1756, in _run_once
                       handle = self._ready.popleft()
                   IndexError: pop from an empty deque

                   During handling of the above exception, another exception occurred:

                   Traceback (most recent call last):
                     File "/var/cowait/cowait/worker/executor.py", line 85, in execute
                       await node.parent.send_done(result, result_type.describe())
                     File "/var/cowait/cowait/worker/parent_client.py", line 86, in send_done
                       await self.msg(TASK_RETURN, result=result, result_type=result_type)
                     File "/var/cowait/cowait/worker/parent_client.py", line 47, in msg
                       **msg,
                     File "/var/cowait/cowait/worker/parent_client.py", line 54, in send
                       await self.io.create_task(super().send(msg))
                     File "/opt/conda/lib/python3.7/asyncio/futures.py", line 260, in __await__
                       yield self  # This tells Task to wait for completion.
                   concurrent.futures._base.CancelledError

@johanhenriksson What do you make of this?

Should be good to go, I'll fix some larger data sets (30gb+) on s3 so we can actually leverage the full power of vaex. I don't think this setup with reading/writing to disk a lot is suitable for our kubernetes setup atm.
Bonus 🎉
UI screencast of this job in action: https://recordit.co/eW4aFFYfkM

@maartenbreddels When multiple Fit tasks are run in parallell in docker, each tasks thinks it has access to the full allowed memory set by your docker settings. If I limit the max memory the containers can use (docker run --memory), vaex still thinks it has access to the full memory of the host machine, and as a result, all parallel tasks OOM when the combined used memory exceeds what's available on the host.
Is there a way to set the maximum amount of memory vaex used by vaex? I imagine something similar to --memory-limit in dask.


Use sandbox-exec on macOS to disable writes to TEA_PREFIX

Runs the repl or commands inside a sandboxed environment.
The sandbox allows everything by default, and then disables writes to all files under TEA_PREFIX.
Fixes #77

edit: ok, there's quite a few tests at the moment that expect TEA_PREFIX to be writeable 😅

Example:
$ ~/s/t/cli (sandbox-exec)> tea run -S +deno.land
tea: pantries sync’d ⎷
this is a temporary shell containing the following packages:
<EMAIL_ADDRESS>when done type: `exit'
tea %~ touch $TEA_PREFIX/what
touch: /Users/gustav/.tea/what: Operation not permitted
tea %~ touch ~/what
tea %~

#77 actually wanted the opposite (wording was poor, my bad 😓), restricting writes only to TEA_PREFIX.
I think though I was thinking about “builds” when making that ticket, and well, we have since decided that users won't do builds. However it would still be a welcome change to @teaxyz/pantry-core’s build.ts.
For the REPL, I'm not sure we should sandbox that either.
open to a discussion about what should be sandboxed.
I think it could be amazing to sandbox tea-scripts.

I've been noodling this and it's possible this is totally the right time to add this. Where packages must specify what paths in their own prefix they allow writes. And we figure out the issues with the tests.
Most packages should not allow writes in $TEA_PREFIX. This is a nice feature.


Port behaviour tests to V3 API.

This replaces curl/jq calls with etcdctl, as the V3 API is much more difficult to use via curl.
Fixes #126.

So, not sure how to proceed here - I'm using a locally running etcd for the Debian autopkgtests, and haven't tested the podman stuff. As test/behaviou_test.sh has everything commented out, I proceeded to do so for the other two tests as well, but probably something self-consisten (and/or usable with Github actions or so?) is needed here.
When you use a local etcd you can use the better check for the vip from the bottom of behaviour_test.sh in the other two tests as well instead of grepping, I could add that.


Flymo Easilife 500 HA fixes

This is a cleaned PR that adds the Easilife fixes into the HA component to cope with the slightly different data the Easilife go 500 returns.

cancelling due to changes upstream to the library that makes this get handled in the library instead of the HA component.