OpenTelemetryCollector customisation for AmpAddOn

Describe the feature
AmpAddOn enhancement to enable the user to provide their own OpenTelemetryCollector
Use Case
e.g. configure additional prometheus scrape_configs
Proposed Solution
No response
Other Information
No response
Acknowledgements

[X] I may be able to implement this feature request

CDK version used
2.82.0
EKS Blueprints Version
No response
Node.js Version
v20.4.0
Environment details (OS name and version, etc.)
Mac

This is complete with openTelemetryCollector support is added.


Release Notes for 7-19-16

Release Notes for 7-19-16

Closing PR.  Will submit a new one from a re-forked branch.

Closing PR. Will re-fork and send new PR.


[ConstraintLocator] Allow simplification of constructor member with overloaded base

init calls to redeclared types would end up diagnosed as ambiguity,
so locator simplification needs to account for the fact that "base"
of constructor might be overloaded type reference.
Resolves: rdar://84879566

@swift-ci please clean test


Add options to rewrite namespace delimiter for pulsar sql.

Motivation
Fix #4732
Modifications
Add options to rewrite the namespace delimiter, disable by default
Enable rewrite namespace delimiter can work well with superset:

Does this pull request potentially affect one of the following parts:
If yes was chosen, please highlight the changes

Dependencies (does it add or upgrade a dependency): (no)
The public API: (no)
The schema: (no)
The default values of configurations: (no)
The wire protocol: (no)
The rest endpoints: (no)
The admin cli options: (no)
Anything that affects deployment: (no)

Documentation

Does this pull request introduce a new feature? (no)

@codelipenghui I have encountered this before its because superset doesn't escape these characters correctly.  Its more of a superset problem than pulsar problem.

@codelipenghui keep the default separator as "/" for backward compatibility and allow overwriting with other separator.
@jerrypeng we need this overwritten feature. so +1 for having this feature. but agreed with keeping the default as "/".

@jerrypeng @sijie
I have change the default delimiter to "/",  PTAL.

run cpp tests

@sijie @codelipenghui sure we can have this feature.  Lets add some comments in pulsar.properties for the config because it could be very dangerous and potentially render all queries to fail since tenant and namespace  names can contain all sort of characters e.g. "_" or "-".  If the delimiter is set to a symbol that already exists in the tenant name or namespace name.
 PTAL
Test Result (15 failures / +15)
org.apache.pulsar.sql.presto.TestPulsarMetadata.testGetColumnHandles
org.apache.pulsar.sql.presto.TestPulsarMetadata.testGetTableHandle
org.apache.pulsar.sql.presto.TestPulsarMetadata.testGetTableMetadata
org.apache.pulsar.sql.presto.TestPulsarMetadata.testGetTableMetadataTableBlankSchema
org.apache.pulsar.sql.presto.TestPulsarMetadata.testGetTableMetadataTableInvalidSchema
org.apache.pulsar.sql.presto.TestPulsarMetadata.testGetTableMetadataTableNoSchema
org.apache.pulsar.sql.presto.TestPulsarMetadata.testGetTableMetadataWrongSchema
org.apache.pulsar.sql.presto.TestPulsarMetadata.testGetTableMetadataWrongTable
org.apache.pulsar.sql.presto.TestPulsarMetadata.testListSchemaNames
org.apache.pulsar.sql.presto.TestPulsarMetadata.testListTable

run cpp tests

run cpp tests
run java8 tests

run java8 tests

@codelipenghui @sijie in regards to this merged PR, what happens if the topic name has a "-"? In SQL you will have to delimit that as well with quotes e.g.
select * from public."public/default"."test-topic"
Even if you change the namespace delimiter from "/" to something to prevent needing to use quotes to delimit, you will still run into the issue with the topic name.

mark this as 2.4.1 for conflict #4779


Stream Analytics Output to Azure Function

Hi,
Would it be possible to clarify exactly what is supported as an Azure Function for the Output of Stream Analytics. From this document, it seems that using an Azure Function that is Virtual Network Integrated is not supported. Does that mean that only Consumption Azure Functions are supported.
How does Stream Analytics access Azure Function from a Network Perspective? What needs to be done for the Consumption Azure Function to be reachable from Stream Analytics.
Thank you in advance!
[Enter feedback here]

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: d4ee2063-bbb7-3b7f-5692-ff876fcf8260
Version Independent ID: 95d1c2a5-2b22-d2e8-3e32-6008cd82f8ec
Content: Tutorial - Run Azure Functions in Azure Stream Analytics jobs
Content Source: articles/stream-analytics/stream-analytics-with-azure-functions.md
Service: stream-analytics
GitHub Login: @enkrumah
Microsoft Alias: ebnkruma

@XavierGuerette Thanks for the question. GitHub forum is dedicated for docs related issues. For any technical queries or clarifications, we encourage to utilize Microsoft Q & A platform. Kindly raise your query on Microsoft Q&A Platform.

@sidramadoss Could you please help clarify on the above customer ask? Thank you

Discussion is taken offline.
I'll leave the issue open until we clarify the ask and can update the doc accordingly.

Surfacing the result of our offline conversation here and closing the issue.
There are 3 elements here:

How ASA supports private networking (private endpoints in the cluster SKU, which include functions), answered by the doc
How Function supports private networking depending on the SKU, answered by their doc
How ASA and Function are layered together in the architecture, on the data plane and the networking plane

Both products have constraints that prevent every possible permutation to be designed (direct output from ASA to function via a VNET). But by using event hub as an intermediary layer, every scenario should be covered (output from ASA to event hub then consumed by function).
#please-close


Make esp32 temperature measurement app commissionable

Problem
DevkitC of temperature measurement app reports being a thread device (does not use the new network config apis)
Change overview
Use the new network config API (logic copied from the all-clusters-app for esp32)
Testing
Flashed a devkit-c and commissioned. Without this, it fails (says wifi cretendials provided but thread device reported), after the change it passes.

Fast tracking platform changes to unblock testing.


Dev/jeffkl/directory build props import tests#806

Hi @John1241, I'm your friendly neighborhood .NET Foundation Pull Request Bot (You can call me DNFBOT). Thanks for your contribution!

In order for us to evaluate and accept your PR, we ask that you sign a contribution license agreement. It's all electronic and will take just minutes. I promise there's no faxing. https://cla2.dotnetfoundation.org.

TTYL, DNFBOT;

Not sure if this was sent by mistake?  Let me know what you're trying to do and I'll help out.


Calling a workflow from within itself using Hera to avoid argo workflows crash for very large workflows (20,000+ pods)

Hi team,
if I try to use w.to_yaml inside the Workflow context to create a task to submit the workflow, it doesn't work.
We have a backfill that we want to run, but even with parallelism set to a smaller value, the overall number of tasks crashes the argo workflows server hence we came up with this method, but this is not working either,
Any suggestions ?

w.to_yaml() isn't possible before the context is closed (it cannot create a yml before you've finished declaring the workflow).
Have you looked into cron-backfills in argo? We do support with_sequence in Hera.


EXLM-83: accordion block

Please always provide the Jira Issue your PR is for, as well as test URLs where your change can be observed (before and after):
Jira ID: EXLM-83
Test URLs:

Before: https://main--exlm--adobe-experience-league.hlx.page/docs/integrations-learn/experience-cloud/solution-categories/content-management
After: https://feature-exlm-83-accordion--exlm--adobe-experience-league.hlx.page/docs/integrations-learn/experience-cloud/solution-categories/content-management

@gandal-adobe LGTM


update(workbox-lab): update to Workbox v3

This commit updates the starter and solution code to Workbox v3. It also
cleans up the HTML and CSS.

resolves #16


Add support for notification service plugins

The PRs show that there is a clear demand for other notification services beside email. We have Telegram, Slack, Pushbullet, Apprise and Discord. I do not want to maintain those and consider them out of scope for snapraid-runner. It might be a good idea to add a simple, configurable hook-in point with well-defined API so others can easily provide plugins for their favorite notification service.

I haven't looked into it too deeply, but I believe Apprise is a single configurable hook-in point, perhaps with a well-defined API for different notification services.

If you can integrate apprise, it will take care of pretty much of notification needs: https://github.com/caronc/apprise
Example of a tool using it: https://github.com/dgtlmoon/changedetection.io

So I've whipped up a quick and dirty fork with apprise support if there's still any interest https://github.com/superjkl/snapraid-runner


feat(src): pasting clause uses new uuid - I88

Issue #88
~Create new uuid for a clause that was copy/pasted~
Handle html on a copy/paste and abstract out clause paste logic to Cicero-UI
Changes

~On a paste event, we take the Document on the event and make it mutable in order to change the clauseid to a uuid and revert everything to immutable~
On a paste event, we handle html type in the paste effect

Flags

NOTE: This functionality may need to be moved to @accordproject/cicero-ui to abstract out clause template specific logic.

This could take the form of replicating the logic in cicero-ui, passing it down, and running props.onPaste if it has been passed down

This has the issue of onPaste requiring PluginManager and FromHTML from markdown-editor, which means they will need to be imported into cicero-ui only for them to be passed down into markdown-editor from which they came from originally

Another form this could take is a higher order function, where the clause specific logic is passed down and used in onPaste if it exists and then carries on to the rest of the plugin process.
if (transfer.type !== 'html') return undefined; may need to be adjusted with next() or something similar? Unsure what next() really does here.

Slate uses IMMUTABLE.js, so we need to augment multiple layers to be mutable in order to then change the clauseid and then augment everything back to immutable.
Further work is needed in the redux store of TSv2, showcased here where the new uuid's placed onto the clauses are not represented in the store, and moreover the extra pasted clause does not appear at all.

DOM:

Store:

Related Issues

TSv2 Issue 85
Cicero UI Issue 134

Yes, the clause specific logic should be in Cicero-ui not here. I think we can register multiple onPaste handlers on the underlying slate editor, they are chained by the call to next() I believe. We also need to test what happens on paste of html that contains an existing clause div.

Yes, the clause specific logic should be in Cicero-ui not here. I think we can register multiple onPaste handlers on the underlying slate editor, they are chained by the call to next() I believe. We also need to test what happens on paste of html that contains an existing clause div.

Not sure I understand how to go about this. This is the direction I am currently going in:
markdown-editor inside SlateAsInputEditor:
const onPaste = (event, editor, next) => {
    const pluginManager = new PluginManager(props.plugins);

    if (isEditable(editor, 'paste')) {
      const transfer = getEventTransfer(event);
      const mutableFragment = transfer.fragment.asMutable();
      const mutableNodes = mutableFragment.nodes.asMutable();
      pluginManager.onPaste(mutableNodes);

      mutableFragment.nodes = mutableNodes.asImmutable();
      transfer.fragment = mutableFragment.asImmutable();
      editor.insertFragment(transfer.fragment);

      if (transfer.type !== 'html') return undefined;

      const fromHtml = new FromHTML(pluginManager);
      const { document } = fromHtml.convert(editor, transfer.html);
      editor.insertFragment(document);
      return undefined;
    }

    return false;
  };

cicero-ui in ClausePlugin:
const onPaste = (nodes) => {
   // nodes is a Class, and errors when trying to run map on the following line
    nodes.map((node) => {
      if (isHeadingClause(node)) {
        const mutableNode = node.asMutable();
        const mutableDataMap = mutableNode.data.asMutable();
        const mutableAttributesMap = mutableDataMap.get('attributes');

        mutableAttributesMap.clauseid = uuidv4(); // unique identifier for a clause instance

        mutableDataMap.set('attributes', mutableAttributesMap);
        mutableNode.data = mutableDataMap.asImmutable();
        return mutableNode;
      }
      return node;
    });
  };

Further issues should consider:

Handling different transfer.type situations

Editor instance (Slate) to another editor instance (Slate)
Raw HTML (Chrome to VSCode to Chrome)
Editor instance (Slate) to Google Docs and back (recognize <clause> formatting)

Ensure onPaste handlers are not doubling up work for any transfer.type cases

@accordproject/cicero-ui/src/plugins/ClausePlugin
@accordproject/markdown-editor/src/SlateAsInputEditor

What fragment.type results when copying pasting from one editor to another instance of an editor?

fragment as a transfer.type is a Slate DOM block fragment, not HTML anymore, so copying and pasting from the same editor into itself should be a fragment
However, copying pasting from one editor to another instance of an editor may result in HTML or fragment, needs experimentation

Preserve formatting, if H1 in HTML, we should be preserving the formatting

Slate has a mechanism for mapping from HTML tags to Slate blocks here


Remove torchffi import

Looks like it's not there anymore :) Feel free to reopen if I missed it. Thanks Ivo!


feat: implement EOF methods in Inspector trait

Motivation
The Inspector trait in revm includes now the eofcreate and eofcreate_end methods with default implementations, we need to add custom implementations for each type implementing the trait
Solution
The implementations aim to have the same behaviour as the existing create and create_end methods.

ok then what is missing in this PR, how can we move it forward? or should we close it? @DaniPopes @mattsse @rakita

We're waiting on a revm release

ok, updated for latest changes in revm, ptal @DaniPopes @mattsse @rakita


add functioncode 0x10

This adds functioncode 0x10 write multiple registers functionality.
Tested only on SDM120 to change the settings of the meter, writing 2 parameters at once as described in the manual. Unfortunatly I have no slave devices available to test writes with different data length.
EDIT writing two parameters on SDM as tested is writing 4 bytes.

Thank you for the addition! I'll google for sample data to write a test case.


Python 3 compatibility?

@ vfilimonov are you perhaps planning to do this at some stage?

I would be happy to accept a PR, but I've never used Python 3 to do this
and perform tests by myself
On 15 Mar 2016, at 08:54, ceaza<EMAIL_ADDRESS>wrote:
@ vfilimonov are you perhaps planning to do this at some stage?
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/vfilimonov/pydatastream/issues/3

Hi Vladimir
Made some simple changes and seems to work in both 3.5 and 2.7.
Python 3.5 dependency becomes suds-py3
All the example items you provide seem to work.
This statement in 3.5 causes warning but this seems to be a spyder issue:
http://stackoverflow.com/questions/30519487/pandas-error-invalid-value-encountered

Performing several requests at once
r1 = '@AAPL~OHLCV~2013-11-26~D'
r2 = 'U:MMM~=P,MV,PO~2013-11-26~D'
res = DWE.request_many([r1,r2])

print(DWE.parse_record(res[0]))
print(DWE.parse_record(res[1]))

/pydatastream.py:201: UserWarning: [DWE] FAILED: $$"ER", E100, INVALID CODE

OR EXPRESSION ENTERED, xxxxxx(PH)
warnings.warn('[DWE] ' + status['StatusMessage'])
/pydatastream.py:201: UserWarning: [DWE] FAILED: $$"ER", E100, INVALID
CODE OR EXPRESSION ENTERED, S&PCOMP(P)
warnings.warn('[DWE] ' + status['StatusMessage'])
C:\Python35\WinPython-64bit-<IP_ADDRESS>\python-3.5.1.amd64\lib\site-packages\pandas\core\format.py:2087:
RuntimeWarning: invalid value encountered in greater
has_large_values = (abs_vals > 1e8).any()
C:\Python35\WinPython-64bit-<IP_ADDRESS>\python-3.5.1.amd64\lib\site-packages\pandas\core\format.py:2088:
RuntimeWarning: invalid value encountered in less
has_small_values = ((abs_vals < 10 ** (-self.digits)) &
C:\Python35\WinPython-64bit-<IP_ADDRESS>\python-3.5.1.amd64\lib\site-packages\pandas\core\format.py:2089:
RuntimeWarning: invalid value encountered in greater
(abs_vals > 0)).any()

Thanks for the package.
Charles
On Tue, Mar 15, 2016 at 4:20 PM, Vladimir Filimonov <
<EMAIL_ADDRESS>wrote:

I would be happy to accept a PR, but I've never used Python 3 to do this
and perform tests by myself
On 15 Mar 2016, at 08:54, ceaza<EMAIL_ADDRESS>wrote:
@ vfilimonov are you perhaps planning to do this at some stage?
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/vfilimonov/pydatastream/issues/3
—
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub:
https://github.com/vfilimonov/pydatastream/issues/3#issuecomment-196842606

import pandas as pd
import datetime as dt
import warnings
from suds.client import Client
TODO: RequestRecordAsXML is more efficient than RequestRecord as it does not return
datatypes for each value (thus response is ~2 times smaller)
WSDL_URL = 'http://dataworks.thomson.com/Dataworks/Enterprise/1.0/webserviceclient.asmx?WSDL'
class DatastreamException(Exception):
pass
class Datastream:
def init(self, username, password, url=WSDL_URL):
"""Creating connection to the Thomson Reuters Dataworks Enterprise (DWE) server
(former Thomson Reuters Datastream).
"""
self.client = Client(url, username=username, password=password)
    self.show_request = False   # If True then request string will be printed
    self.last_status = None     # Will contain status of last request
    self.raise_on_error = True  # if True then error request will raise, otherwise
                                # either empty dataframe or partially retrieved
                                # data will be returned

    # Trying to connect
    try:
        self.ver = self.version()
    except:
        raise DatastreamException('Can not retrieve the data')

    # Creating UserData object
    self.userdata = self.client.factory.create('UserData')
    self.userdata.Username = username
    self.userdata.Password = password

    # Check available data sources
    if 'Datastream' not in self.sources():
        warnings.warn("'Datastream' source is not available for given subscription!")

@staticmethod
def info():
    print('Datastream Navigator:')
    print('http://product.datastream.com/navigator/')
    print('')
    print('Datastream documentation:')
    print('http://dtg.tfn.com/data/DataStream.html')
    print('')
    print('Dataworks Enterprise documentation:')
    print('http://dataworks.thomson.com/Dataworks/Enterprise/1.0/')
    print('')
    print('Thomson Reuters Datastream support:')
    print('https://customers.reuters.com/sc/Contactus/simple?product=Datastream&env=PU&TP=Y')

def version(self):
    """Return version of the TR DWE."""
    res = self.client.service.Version()
    return '.'.join([str(x) for x in res[0]])

def system_info(self):
    """Return system information."""
    res = self.client.service.SystemInfo()
    res = {str(x[0]): x[1] for x in res[0]}

    to_str = lambda arr: '.'.join([str(x) for x in arr[0]])
    res['OSVersion'] = to_str(res['OSVersion'])
    res['RuntimeVersion'] = to_str(res['RuntimeVersion'])
    res['Version'] = to_str(res['Version'])

    res['Name'] = str(res['Name'])
    res['Server'] = str(res['Server'])
    res['LocalNameCheck'] = str(res['LocalNameCheck'])
    res['UserHostAddress'] = str(res['UserHostAddress'])

    return res

def sources(self):
    """Return available sources of data."""
    res = self.client.service.Sources(self.userdata, 0)
    return [str(x[0]) for x in res[0]]

def request(self, query, source='Datastream',
            fields=None, options=None, symbol_set=None, tag=None):
    """General function to retrieve one record in raw format.

       query - query string for DWE system. This may be a simple instrument name
               or more complicated request. Refer to the documentation for the
               format.
       source - The name of datasource (default: "Datastream")
       fields - Fields to be retrieved (used when the requester does not want all
                fields to be delivered).
       options - Options for specific data source. Many of datasources do not require
                 opptions string. Refer to the documentation of the specific
                 datasource for allowed syntax.
       symbol_set - The symbol set used inside the instrument (used for mapping
                    identifiers within the request. Refer to the documentation for
                    the details.
       tag - User-defined cookie that can be used to match up requests and response.
             It will be returned back in the response. The string should not be
             longer than 256 characters.
    """
    if self.show_request:
        print('Request:', query)

    rd = self.client.factory.create('RequestData')
    rd.Source = source
    rd.Instrument = query
    if fields is not None:
        rd.Fields = self.client.factory.create('ArrayOfString')
        rd.Fields.string = fields
    rd.SymbolSet = symbol_set
    rd.Options = options
    rd.Tag = tag

    return self.client.service.RequestRecord(self.userdata, rd, 0)

def request_many(self, queries, source='Datastream',
                 fields=None, options=None, symbol_set=None, tag=None):
    """General function to retrieve one record in raw format.

       query - list of query strings for DWE system.
       source - The name of datasource (default: "Datastream")
       fields - Fields to be retrieved (used when the requester does not want all
                fields to be delivered).
       options - Options for specific data source. Many of datasources do not require
                 opptions string. Refer to the documentation of the specific
                 datasource for allowed syntax.
       symbol_set - The symbol set used inside the instrument (used for mapping
                    identifiers within the request. Refer to the documentation for
                    the details.
       tag - User-defined cookie that can be used to match up requests and response.
             It will be returned back in the response. The string should not be
             longer than 256 characters.
       NB! source, options, symbol_set and tag are assumed to be identical for all
           requests in the list
    """
    if self.show_request:
        print('Requests:', queries)

    if not isinstance(queries, list):
        queries = [queries]

    req = self.client.factory.create('ArrayOfRequestData')
    req.RequestData = []
    for q in queries:
        rd = self.client.factory.create('RequestData')
        rd.Source = source
        rd.Instrument = q
        if fields is not None:
            rd.Fields = self.client.factory.create('ArrayOfString')
            rd.Fields.string = fields
        rd.SymbolSet = symbol_set
        rd.Options = options
        rd.Tag = tag

        req.RequestData.append(rd)

    return self.client.service.RequestRecords(self.userdata, req, 0)[0]

#################################################################################
def status(self, record=None):
    """Extract status from the retrieved data and save it as a property of an object.
       If record with data is not specified then the status of previous operation is
       returned.

       status - dictionary with data source, string with request and status type,
                code and message.

       status['StatusType']: 'Connected' - the data is fine
                             'Stale'     - the source is unavailable. It may be
                                           worthwhile to try again later
                             'Failure'   - data could not be obtained (e.g. the
                                           instrument is incorrect)
                             'Pending'   - for internal use only
       status['StatusCode']: 0 - 'No Error'
                             1 - 'Disconnected'
                             2 - 'Source Fault'
                             3 - 'Network Fault'
                             4 - 'Access Denied' (user does not have permissions)
                             5 - 'No Such Item' (no instrument with given name)
                             11 - 'Blocking Timeout'
                             12 - 'Internal'
    """
    if record is not None:
        self.last_status = {'Source': str(record['Source']),
                            'StatusType': str(record['StatusType']),
                            'StatusCode': record['StatusCode'],
                            'StatusMessage': str(record['StatusMessage']),
                            'Request': str(record['Instrument'])}
    return self.last_status

def _test_status_and_warn(self):
    """Test status of last request and post warning if necessary.
    """
    status = self.last_status
    if status['StatusType'] != 'Connected':
        if isinstance(status['StatusMessage'], (str)):
            warnings.warn('[DWE] ' + status['StatusMessage'])
        elif isinstance(status['StatusMessage'], list):
            warnings.warn('[DWE] ' + ';'.join(status['StatusMessage']))

#################################################################################
@staticmethod
def extract_data(raw):
    """Extracts data from the raw response and returns it as a dictionary."""
    return {x[0]: x[1] for x in raw['Fields'][0]}

def parse_record(self, raw, indx=0):
    """Parse raw data (that is retrieved by "request") and return pandas.DataFrame.
       Returns tuple (data, metadata)

       data - pandas.DataFrame with retrieved data.
       metadata - pandas.DataFrame with info about symbol, currency, frequency,
                  displayname and status of given request
    """
    suffix = '' if indx == 0 else '_%i' % (indx+1)

    # Parsing status
    status = self.status(raw)

    # Testing if no errors
    if status['StatusType'] != 'Connected':
        if self.raise_on_error:
            raise DatastreamException('%s (error %i): %s --> "%s"' %
                                      (status['StatusType'], status['StatusCode'],
                                       status['StatusMessage'], status['Request']))
        else:
            self._test_status_and_warn()
            return pd.DataFrame(), {}

    record = self.extract_data(raw)
    get_field = lambda fldname: record[fldname+suffix]

    try:
        error = get_field('INSTERROR')
        if self.raise_on_error:
            raise DatastreamException('Error: %s --> "%s"' %
                                      (error, status['Request']))
        else:
            self.last_status['StatusMessage'] = error
            self.last_status['StatusType'] = 'INSTERROR'
            self._test_status_and_warn()
            metadata = {'Frequency': '', 'Currency': '', 'DisplayName': '',
                        'Symbol': '', 'Status': error}
    except KeyError:
        # Parsing metadata of the symbol
        # NB! currency might be returned as symbol thus "unicode" should be used
        metadata = {'Frequency': str(get_field('FREQUENCY')),
                    'Currency': str(get_field('CCY')),
                    'DisplayName': str(get_field('DISPNAME')),
                    'Symbol': str(get_field('SYMBOL')),
                    'Status': 'OK'}

    # Fields with data
    if suffix == '':
        fields = [str(x) for x in record if '_' not in x]
    else:
        fields = [str(x) for x in record if suffix in x]

    # Filter metadata
    meta_fields = ['CCY', 'DISPNAME', 'FREQUENCY', 'SYMBOL', 'DATE', 'INSTERROR']
    fields = [x.replace(suffix, '') for x in fields
              if not any([y in x for y in meta_fields])]

    if 'DATE'+suffix in record:
        date = record['DATE'+suffix]
    elif 'DATE' in record:
        date = record['DATE']
    else:
        date = None

    if len(fields) > 0 and date is not None:
        # Check if we have a single value or a series
        if isinstance(date, dt.datetime):
            data = pd.DataFrame({x: [get_field(x)] for x in fields},
                                index=[date])
        else:
            data = pd.DataFrame({x: get_field(x)[0] for x in fields},
                                index=date[0])
    else:
        data = pd.DataFrame()

    metadata = pd.DataFrame(metadata, index=[indx])
    metadata = metadata[['Symbol', 'DisplayName', 'Currency', 'Frequency', 'Status']]
    return data, metadata

def parse_record_static(self, raw):
    """Parse raw data (that is retrieved by static request) and return pandas.DataFrame.
       Returns tuple (data, metadata)

       data - pandas.DataFrame with retrieved data.
       metadata - pandas.DataFrame with info about symbol, currency, frequency,
                  displayname and status of given request
    """
    # Parsing status
    status = self.status(raw)

    # Testing if no errors
    if status['StatusType'] != 'Connected':
        if self.raise_on_error:
            raise DatastreamException('%s (error %i): %s --> "%s"' %
                                      (status['StatusType'], status['StatusCode'],
                                       status['StatusMessage'], status['Request']))
        else:
            self._test_status_and_warn()
            return pd.DataFrame(), {}

    # Convert record to dict
    record = self.extract_data(raw)

    try:
        error = record['INSTERROR']
        if self.raise_on_error:
            raise DatastreamException('Error: %s --> "%s"' %
                                      (error, status['Request']))
        else:
            self.last_status['StatusMessage'] = error
            self.last_status['StatusType'] = 'INSTERROR'
            self._test_status_and_warn()
            return pd.DataFrame(), {'Status': error, 'Date': None}
    except KeyError:
        metadata = {'Status': 'OK', 'Date': ''}

    # All fields that are available
    fields = [x for x in record if '_' not in x]
    metadata['Date'] = record['DATE']
    fields.remove('DATE')

    # Number of elements
    num = len([x[0] for x in record if 'SYMBOL' in x])

    # field naming 'CCY', 'CCY_2', 'CCY_3', ...
    fld_name = lambda field, indx: field if indx == 0 else field + '_%i' % (indx+1)

    # Construct pd.DataFrame
    res = pd.DataFrame({fld: [record[fld_name(fld, ind)]
                              if fld_name(fld, ind) in record else ''
                              for ind in range(num)]
                        for fld in fields})
    return res, metadata

#################################################################################
@staticmethod
def construct_request(ticker, fields=None, date=None,
                      date_from=None, date_to=None, freq=None):
    """Construct a request string for querying TR DWE.

       tickers - ticker or symbol
       fields  - list of fields.
       date    - date for a single-date query
       date_from, date_to - date range (used only if "date" is not specified)
       freq    - frequency of data: daily('D'), weekly('W') or monthly('M')
                 Use here 'REP' for static requests

       Some of available fields:
       P  - adjusted closing price
       PO - opening price
       PH - high price
       PL - low price
       VO - volume, which is expressed in 1000's of shares.
       UP - unadjusted price
       OI - open interest

       MV - market value
       EPS - earnings per share
       DI - dividend index
       MTVB - market to book value
       PTVB - price to book value
       ...

       The full list of data fields is available at http://dtg.tfn.com/.
    """
    if isinstance(ticker, list):
        request = ','.join(ticker)
    else:
        request = ticker
    if fields is not None:
        if isinstance(fields, (str)):
            request += '~=' + fields
        elif isinstance(fields, list) and len(fields) > 0:
            request += '~=' + ','.join(fields)
    if date is not None:
        request += '~@' + pd.to_datetime(date).strftime('%Y-%m-%d')
    else:
        if date_from is not None:
            request += '~' + pd.to_datetime(date_from).strftime('%Y-%m-%d')
        if date_to is not None:
            request += '~:' + pd.to_datetime(date_to).strftime('%Y-%m-%d')
    if freq is not None:
        request += '~' + freq
    return request

#################################################################################
def fetch(self, tickers, fields=None, date=None, date_from=None, date_to=None,
          freq='D', only_data=True, static=False):
    """Fetch data from TR DWE.

       tickers - ticker or list of tickers
       fields  - list of fields.
       date    - date for a single-date query
       date_from, date_to - date range (used only if "date" is not specified)
       freq    - frequency of data: daily('D'), weekly('W') or monthly('M')
       only_data - if True then metadata will not be returned
       static  - if True "static" request is created (i.e. not a series).
                 In this case 'date_from', 'date_to' and 'freq' are ignored

       NB! in case list of tickers is requested, pandas.Panel is returned.

       Some of available fields:
       P  - adjusted closing price
       PO - opening price
       PH - high price
       PL - low price
       VO - volume, which is expressed in 1000's of shares.
       UP - unadjusted price
       OI - open interest

       MV - market value
       EPS - earnings per share
       DI - dividend index
       MTVB - market to book value
       PTVB - price to book value
       ...

       The full list of data fields is available at http://dtg.tfn.com/.
    """
    if static:
        query = self.construct_request(tickers, fields, date, freq='REP')
    else:
        query = self.construct_request(tickers, fields, date, date_from, date_to, freq)

    raw = self.request(query)

    if static:
        data, metadata = self.parse_record_static(raw)
    elif isinstance(tickers, (str)) or len(tickers) == 1:
        data, metadata = self.parse_record(raw)
    elif isinstance(tickers, list):
        metadata = pd.DataFrame()
        data = {}
        for indx in range(len(tickers)):
            dat, meta = self.parse_record(raw, indx)
            data[tickers[indx]] = dat
            metadata = metadata.append(meta, ignore_index=False)

        data = pd.Panel(data).swapaxes('items', 'minor')
    else:
        raise DatastreamException(('First argument should be either ticker or '
                                   'list of tickers'))

    if only_data:
        return data
    else:
        return data, metadata

#################################################################################
def get_OHLCV(self, ticker, date=None, date_from=None, date_to=None):
    """Get Open, High, Low, Close prices and daily Volume for a given ticker.

       ticker  - ticker or symbol
       date    - date for a single-date query
       date_from, date_to - date range (used only if "date" is not specified)

       Returns pandas.Dataframe with data. If error occurs, then it is printed as
       a warning.
    """
    data, meta = self.fetch(ticker + "~OHLCV", None, date,
                            date_from, date_to, 'D', only_data=False)
    return data

def get_OHLC(self, ticker, date=None, date_from=None, date_to=None):
    """Get Open, High, Low and Close prices for a given ticker.

       ticker  - ticker or symbol
       date    - date for a single-date query
       date_from, date_to - date range (used only if "date" is not specified)

       Returns pandas.Dataframe with data. If error occurs, then it is printed as
       a warning.
    """
    data, meta = self.fetch(ticker + "~OHLC", None, date,
                            date_from, date_to, 'D', only_data=False)
    return data

def get_price(self, ticker, date=None, date_from=None, date_to=None):
    """Get Close price for a given ticker.

       ticker  - ticker or symbol
       date    - date for a single-date query
       date_from, date_to - date range (used only if "date" is not specified)

       Returns pandas.Dataframe with data. If error occurs, then it is printed as
       a warning.
    """
    data, meta = self.fetch(ticker, None, date,
                            date_from, date_to, 'D', only_data=False)
    return data

#################################################################################
def get_constituents(self, index_ticker, date=None, return_raw=False):
    """ Get a list of all constituents of a given index.

        index_ticker - Datastream ticker for index
        date         - date for which list should be retrieved (if None then
                       list of present constituents is retrieved)
        return_raw   - Method does not parse the response to pd.DataFrame format
                       and returns the raw dict (for debugging purposes)
    """
    if date is not None:
        str_date = pd.to_datetime(date).strftime('%m%y')
    else:
        str_date = ''
    # Note: ~XREF is equal to the following large request
    # ~REP~=DSCD,EXMNEM,GEOG,GEOGC,IBTKR,INDC,INDG,INDM,INDX,INDXEG,INDXFS,INDXL,
    #       INDXS,ISIN,ISINID,LOC,MNEM,NAME,SECD,TYPE
    query = 'L' + index_ticker + str_date + '~XREF'
    raw = self.request(query)

    if return_raw:
        return self.extract_data(raw)

    res, metadata = self.parse_record_static(raw)
    return res

if name == "main":
DWE = Datastream(username="", password="")
print(DWE.system_info())

print(DWE.sources())

data = DWE.get_price('@AAPL', date='2000-05-03')
print(data)
data = DWE.get_price('@AAPL', date_from='2008', date_to='2009')
print(data.head())
data.plot()
print(data.resample('M', how='last'))
data = DWE.get_OHLCV('@AAPL', date_from='2008', date_to='2009')
data = DWE.get_OHLC('S&PCOMP', date_from='May 6, 2010')
data = DWE.fetch('@AAPL', ['P','MV','VO',], date_from='2000-01-01')

Requesting several tickers at once
res = DWE.fetch(['@AAPL','U:MMM'], fields=['P','MV','VO','PH'], date_from='2000-05-03')
print(res['MV'].head())
DWE.raise_on_error = False
res = DWE.fetch(['@AAPL','U:MMM','xxxxxx','S&PCOMP'], fields=['P','MV','VO','PH'], date_from='2000-05-03')

print(res['MV'].head())
print(res['P'].head())

Constituents list for indices
res = DWE.get_constituents('S&PCOMP')
print(res.ix[0])
res = DWE.get_constituents('S&PCOMP', '1-sept-2013')
print(res.ix[0])

Static requests
res = DWE.fetch(['D:BAS','D:BASX','HN:BAS','I:BAF','BFA','@BFFAF','S:BAS'],
                ['ISIN', 'ISINID', 'NAME'], static=True)
print(res)
res = DWE.fetch(['D:BAS','D:BASX','HN:BAS','I:BAF','BFA','@BFFAF','S:BAS'],
                ['P', 'MV', 'VO'], static=True)
print(res)

Using custom requests
raw = DWE.request('@AAPL~=P,MV,VO,PH~2013-01-01~D')
print(raw['StatusType'])
print(raw['StatusCode'])

data = DWE.extract_data(raw)
print(data['CCY'])
print(data['MV'])

Get performance information of a particular stock
res = DWE.fetch('@AAPL~PERF', date_from='2011-09-01')
print(res.head())

Get some reference information on a security with "~XREF", including ISIN, industry, etc.
res = DWE.request('U:IBM~XREF')
print(DWE.extract_data(res))

Convert the currency e.g. to Euro with "~~EUR"
res = DWE.fetch('U:IBM(P)~~EUR', date_from='2013-09-01')
print(res.head())

Calculate moving average on 20 days
res = DWE.fetch('MAV#(U:IBM,20D)', date_from='2013-09-01')
print(res.head())

Performing several requests at once
r1 = '@AAPL~OHLCV~2013-11-26~D'
r2 = 'U:MMM~=P,MV,PO~2013-11-26~D'
res = DWE.request_many([r1,r2])

print(DWE.parse_record(res[0]))
print(DWE.parse_record(res[1]))

PR is submitted, my first ever so hope all ok.

Thanks a lot!

BTW, I've seen some commits ([1], [2]) that just correct typos in the previous one.
A good practice here is not to multiply unnecessary entries, but to "amend" a previous one - effectively rewriting it (which will work fine if someone has not pulled it after you have pushed it to remote, for local branches/repos it is rather safe). Here's how you can do this in command line. I'm using SourceTree where it is just a checkbox when you commit.

Sorry about typos and thanks for the tips.
On Thu, Mar 17, 2016 at 12:29 AM, Vladimir Filimonov <
<EMAIL_ADDRESS>wrote:

BTW, I've seen some commits ([1]
https://github.com/vfilimonov/pydatastream/pull/4/commits/66b3817dd91e36c0450c547148fa71848fb68374,
[2]
https://github.com/vfilimonov/pydatastream/pull/4/commits/4baba04841676837b1524f3cee5547b187559c75)
that just correct typos in the previous one.
A good practice here is not to multiply unnecessary entries, but to
"amend" a previous one - effectively rewriting it (which will work fine if
someone has not pulled it after you have pushed it to remote, for local
branches/repos it is rather safe). Here's how you can do this in command
line https://nathanhoad.net/git-amend-your-last-commit. I'm using
SourceTree https://www.atlassian.com/software/sourcetree/overview/
where it is just a checkbox when you commit.
—
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub
https://github.com/vfilimonov/pydatastream/issues/3#issuecomment-197585653


Spammed by Kili github actions

Hello everyone,
Due to one of my last PR's, I'm getting spammed by failing integration tests in CI (cf image below), I get like 5 emails per day.
It seems like there is a leak in one of the pipelines

You forked our repo that contains workflows triggered by cron. So your fork also contains those workflows, and if you don't disable workflows in your repo settings, they will also be triggered by the cron.
Deleting your fork should stop the mails coming from github

Cool thanks, didn't know about that ^^


Inactivity Log Out Issue

We have an ongoing issue that several users have voiced.  They're getting logged out of their database with an inactivity error when they're actively working. I'm pretty sure this is related to KTL as I haven't noticed this problem in other Knack databases. Is this something related to the idlewachdogdelay? Is there something I can do to prevent this from happening?

Hi Daniel,
By default, the KTL does nothing when there is no activity.
This is the code that is executed, that you can override to implement your own behavior:
//This is called after there's been no keypress or mouse click for a certain period of time.
//By default it does nothing.
function idleWatchDogTimeout() {
    //Your specific code here...

    //You can reset the idle wd and do something else.
    //ktl.scenes.resetIdleWatchdog();

    //Or force a logout like this:
    //ktl.account.logout();
}

As you can see, everything is commented-out.
You're sure the inactivity setting is OFF?

If not, send me your app's link by email please.
Norm

Actually, I had the link already and did a quick check.  The inactivity timeout is enabled at 30 minutes.
See here:

Just turn that off if you don't want it.
If you want it but just for some roles or users, the KTL can do that easily.  Let me know and I'll show you how.
Norm

Thanks, Norm. I did disable this to see if things improve but I'm not sure this is the cause of the problem. Looking through our logs I'm noticing quite a few "Server Error" logs that may correspond with the folks being kicked out. Is this something that would happen (particularly on an API or Refresh View error)?

This is still happening after disabling the auto inactivity logout. Specifically, the message says "Your login has expired please log back in to continue. " Each instance looks like it's connected to a Knack API or Refresh View error that's being tracked in the system logs.

Dan,
I checked again and the flag is still enabled in your app.  Are you sure you disabled it?
You're right, the errors are caused by a failure in the view refreshes and the API calls.  This happens when your login has expired.  It can be due to an inactivity timeout set by your app, or simply because it's been more than 2 days or 2 weeks (if Remember be is checked) that you've logged-in.
Now to prevent problems, when the KTL encounters the server error types 401 or 403, it will pop the message and kick you out to prevent an infinite loop of future errors and logs.  This is normal and intentional to inform the user about what's happening.
If you want to check if your app has an inactivity timeout, go in the console and type this:
Knack.app.attributes.settings
Click on the black arrow to open the object and you'll see this:

Oh... I refreshed and it seems to be disabled now!

I bet you just forgot to click the Save Settings button at the bottom of the page!!

Yes, I must not have saved the settings. However, from your explanation, I don't think that this is the cause of most of these logouts.  It sounds like I am going to get logged out anytime there's a 401 or a 403 error. Is that correct? Today alone there are 254 of these errors in my system (it's happened to me 6 times) and that can be frustrating when someone is in the middle of doing something.  Is there anything I can do to prevent or at least limit how frequently this is happening?

Sorry, 46 errors today. Not nearly as bad but still something that's causing frustration for my users.

It sounds like I am going to get logged out anytime there's a 401 or a 403 error. Is that correct?
Yes, that's correct.  But since the 401 error comes from Knack (not the KTL), the users will be kicked out anyways.

At least with the KTL, if the Persistence feature is enabled and they were filling a form, when they to log back in all data is reloaded and they won't lose anything.

that can be frustrating when someone is in the middle of doing something
No because as long as they keep clicking, moving the mouse or typing, this won't happen.

BTW, note that you as a developer using the Builder all the time, you'll be kicked-out of your App if the Builder remains without activity.  You always have to remember to trigger some "fake" activity in both the App and the Builder every few minutes.  This is very annoying!

Turning off the inactivity timeout seems to have mostly fixed the problem! I still would like to have some sort of auto-logout if someone doesn't interact with the system for a long time. If I use the idle watchdog can I do this even if a person has multiple tabs open and doesn't interact with one of those tabs for a long while?


tag 1.0.8, bower: no minify, include plugins

You can try it out with bower install konsumer/prefixfree

+1!

Thanks! Sorry for the delay!


One potential way to automatically add the checked=‘on’ property

This is just one way. Or you can try to find a way to pass down the currently selected option to the list and have it just show/hide the checkmark when it renders that option.
Also you'll need to find a way to hide the checkmark on focus

@andreadelrio Ok, I came full circle on this one to how you were attempting it before. I think what was happening is that we just needed to add a different if statement for the checked property if in singleSelection. I've pushed my changes that pulled that logic out of the return statement to make it clearer.
It seems selectedOptions was being passed so it just needs a label check like I think you were attempting. Yes, consumers can pass options with the same label, but they'll get a console error like this one that is explicit from the EuiComboBox itself.


[FEATURE] Enable Azure Dev CLI to deploy AKS-LZA

Is your feature request related to a problem? Please describe.
A well architect AKS environment can be complicated to build. Enter the AKS deployment helper! However how do we make various sample apps available to customers who want to try out AKS and want the quickest option possible?
Describe the solution you'd like
Azure dev CLI makes it easy and quick to get started on Azure rapidly and is developer focused. Leveraging it to enable customers create well architected AKS environment will get more customers using AKS quickly
Describe alternatives you've considered
Using the AKS deploy helper to create the environment. However the dev CLI option will be even simpler albert not as flexible. The dev CLI can also be used to create sample apps in various languages. We will begin with our javascript app
Additional context

:warning: Start by reading this doc to understand how to make a repo azd compatible

we will be using the AKS construction project for the IaC and have the repo pulled in as a submodule.
Here are the required tasks:

[ ] Create a repository in Azure-Samples org that the deployment and application scripts will be created in
[ ] Use this repo as a guide to create the repo: https://github.com/azure-samples/todo-nodejs-mongo-aks. It includes all the folders required to make a repo Dev CLI enabled. According to this the new repo should have the following folders: .devcontainers, .github/workflows, assets (where the pictures in readme will go), src
[ ] Use the content of this folder  to create the code that goes into the infra folder. the AKS-Construction repo should be reference as a submodule in this project.  We need to have a main.bicep and a main.parameters.json file in the infra folder, how the rest of the folder is structured apart from that is up to you. Make your own main.bicep file in the infra folder that uses the aks-construction's main.bicep file by referencing it like i did in this project. The only difference being that instead of listing the parameters like i did in that file as a json object, put those parameters in a fresh  main.parameters.json file. Use the same parameters i used in my json object. Here is an example of what the parameters file will look like
[ ] The src folder should have two folders in it. The mslearn-aks-workshop-ratings-api folder submodule to the ratings app api repo. The mslearn-aks-workshop-ratings-web folder should do the same but submodule to this repo
[ ] The .devcontainers folder should use the same as this folder. Make changes to the upstream project's .devcontainer folder if need be and copy it into the .devcontainer for this project
[ ] the .github/workflows folder should include this pipeline to deploy the infra and this pipeline to deploy the app. This second pipeline will need access to the yaml files in this folder so this repo will need to be cloned as a submodule. If dev cli only allows you to use 1 pipeline, use the first one and we can figure out how to add the second later
[ ] The project will also require an azure.yaml file as seen in the sample repo provided

After all these steps are completed we will then proceed to complete step 2 and 3 highlighted in this doc

Created repo in https://github.com/carmodyquinnms/AKS-Construction-azd
Gordon helped me with the git submodule (kudos!! Gordon was SOO helpful as always!)
Ran into an issue with the firewall in the submodule and had to switch to 0.9.14 branch of aks-construction with the commands provided by Gordon:
cd aks-construction
git checkout 0.9.14
cd ..
git add aks-construction
git commit -m "moved submodule to 0.9.14"
git push

Then the provisioning ran just fine.
Need to get with mosabami and find out what name I should use to move the repo to Azure presumably under https://github.com/azure-samples/

Changing this issue to align with the product's new direction. We are going to be working out of the:

Create a new folder inside https://github.com/Azure/AKS-Landing-Zone-Accelerator/tree/main/Scenarios/AKS-Secure-Baseline-PrivateCluster/Bicep similar to the all-in-1 folder already existing there. this new folder can be called AZD-Deploy
Configure the AZD to leverage https://github.com/Azure/AKS-Landing-Zone-Accelerator/blob/main/Scenarios/AKS-Secure-Baseline-PrivateCluster/Bicep/All-in-One-Bicep/main.bicep to deploy the azure resources but it should set the enablePrivateCluster parameter to false so it doesnt deploy a private cluster
It should deploy the aks store demo using the yaml files available on the internet https://github.com/Azure-Samples/aks-store-demo/blob/main/aks-store-all-in-one.yaml

Changing this issue to align with the product's new direction. We are going to be working out of the:
we will be using our new avm based bicep files to enable deploy with azd
Create a new folder inside https://github.com/Azure/AKS-Landing-Zone-Accelerator/tree/main/Scenarios/AKS-Secure-Baseline-PrivateCluster/Bicep similar to the all-in-1 folder already existing there. this new folder can be called AZD-Deploy
Configure the AZD to leverage https://github.com/Azure/AKS-Landing-Zone-Accelerator/blob/main/Scenarios/AKS-Secure-Baseline-PrivateCluster/Bicep/All-in-One-Bicep/main.bicep to deploy the azure resources but it should set the enablePrivateCluster parameter to false so it doesnt deploy a private cluster
It should deploy the aks store demo using the yaml files available


TFS. Failed to launch testhost. Possible to make it working?

Description
One test run on server resulted in (say) half of tests to be countered. From [my previous post and the answer],(https://github.com/Microsoft/vstest/issues/1979) i found out that once connection b/w vstest and testhost breaks, the other tests do not get run.
Second and 3rd and 4th and so on... test runs fail to launch testshost at all. None of tests are countered in .trx. No .trx files. To make it more clear, at this stage i'm trying to run each test in its own exec with /Tests:NameOfTest specified.
Is it possible to execute some command to make the testhost working?
Expected behavior
Would be great to know how to get testhost working.
Actual behavior
Failed to launch testhost. I guess that this error occured once and keeped on occuring afterwards. Could be that, at some point some obstacle appeared.
Diagnostic logs
The errors are (one followed by another):
TpTrace Error: 0 : ###, 1, 2019/04/08, 11:24:48.478, ####, vstest.console.dll, ProxyOperationManager: Failed to launch testhost :System.AggregateException: One or more errors occurred. (The system cannot find the file specified) ---> System.ComponentModel.Win32Exception: The system cannot find the file specified
   at System.Diagnostics.Process.StartWithCreateProcess(ProcessStartInfo startInfo)
   at System.Diagnostics.Process.Start()
   at Microsoft.VisualStudio.TestPlatform.PlatformAbstractions.ProcessHelper.LaunchProcess(String processPath, String arguments, String workingDirectory, IDictionary`2 envVariables, Action`2 errorCallback, Action`1 exitCallBack)
   at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Hosting.DefaultTestHostManager.LaunchHost(TestProcessStartInfo testHostStartInfo, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Hosting.DefaultTestHostManager.<>c__DisplayClass35_0.<LaunchTestHostAsync>b__0()
   at System.Threading.Tasks.Task`1.InnerInvoke()
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location where exception was thrown ---
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot)
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Hosting.DefaultTestHostManager.LaunchTestHostAsync(TestProcessStartInfo testHostStartInfo, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
   at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Client.ProxyOperationManager.SetupChannel(IEnumerable`1 sources)
---> (Inner Exception #0) System.ComponentModel.Win32Exception (2): The system cannot find the file specified
   at System.Diagnostics.Process.StartWithCreateProcess(ProcessStartInfo startInfo)
   at System.Diagnostics.Process.Start()
   at Microsoft.VisualStudio.TestPlatform.PlatformAbstractions.ProcessHelper.LaunchProcess(String processPath, String arguments, String workingDirectory, IDictionary`2 envVariables, Action`2 errorCallback, Action`1 exitCallBack)
   at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Hosting.DefaultTestHostManager.LaunchHost(TestProcessStartInfo testHostStartInfo, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Hosting.DefaultTestHostManager.<>c__DisplayClass35_0.<LaunchTestHostAsync>b__0()
   at System.Threading.Tasks.Task`1.InnerInvoke()
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location where exception was thrown ---
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot)
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Hosting.DefaultTestHostManager.LaunchTestHostAsync(TestProcessStartInfo testHostStartInfo, CancellationToken cancellationToken)<---

TpTrace Error: 0 : ###, 1, 2019/04/08, 11:24:48.479, ####, vstest.console.dll, ProxyDiscoveryManager.DiscoverTests: Failed to discover tests: Microsoft.VisualStudio.TestPlatform.ObjectModel.TestPlatformException: Failed to launch testhost with error: System.AggregateException: One or more errors occurred. (The system cannot find the file specified) ---> System.ComponentModel.Win32Exception: The system cannot find the file specified
   at System.Diagnostics.Process.StartWithCreateProcess(ProcessStartInfo startInfo)
   at System.Diagnostics.Process.Start()
   at Microsoft.VisualStudio.TestPlatform.PlatformAbstractions.ProcessHelper.LaunchProcess(String processPath, String arguments, String workingDirectory, IDictionary`2 envVariables, Action`2 errorCallback, Action`1 exitCallBack)
   at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Hosting.DefaultTestHostManager.LaunchHost(TestProcessStartInfo testHostStartInfo, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Hosting.DefaultTestHostManager.<>c__DisplayClass35_0.<LaunchTestHostAsync>b__0()
   at System.Threading.Tasks.Task`1.InnerInvoke()
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location where exception was thrown ---
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot)
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Hosting.DefaultTestHostManager.LaunchTestHostAsync(TestProcessStartInfo testHostStartInfo, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
   at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Client.ProxyOperationManager.SetupChannel(IEnumerable`1 sources)
---> (Inner Exception #0) System.ComponentModel.Win32Exception (2): The system cannot find the file specified
   at System.Diagnostics.Process.StartWithCreateProcess(ProcessStartInfo startInfo)
   at System.Diagnostics.Process.Start()
   at Microsoft.VisualStudio.TestPlatform.PlatformAbstractions.ProcessHelper.LaunchProcess(String processPath, String arguments, String workingDirectory, IDictionary`2 envVariables, Action`2 errorCallback, Action`1 exitCallBack)
   at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Hosting.DefaultTestHostManager.LaunchHost(TestProcessStartInfo testHostStartInfo, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Hosting.DefaultTestHostManager.<>c__DisplayClass35_0.<LaunchTestHostAsync>b__0()
   at System.Threading.Tasks.Task`1.InnerInvoke()
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location where exception was thrown ---
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot)
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Hosting.DefaultTestHostManager.LaunchTestHostAsync(TestProcessStartInfo testHostStartInfo, CancellationToken cancellationToken)<---

   at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Client.ProxyOperationManager.SetupChannel(IEnumerable`1 sources)
   at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Client.ProxyDiscoveryManager.DiscoverTests(DiscoveryCriteria discoveryCriteria, ITestDiscoveryEventsHandler2 eventHandler)

Environment
Dotnet.exe was the latest version 2-3 weeks ago.

From the logs it seems that testhost.exe was not even found by vstest.console. Can you share the diagnostics logs, it will contain information from where we were trying to launch testhost process.

@mayankbansal018, thanks, here's some diagnostics:
TpTrace Verbose: 0 : ###, 1, 2019/04/08, 11:24:34.332, ############, vstest.console.dll, DefaultTestHostmanager: Full path of TestHost\testhost.exe is C:\Program Files\dotnet\sdk\2.1.503\Extensions\..//TestHost\testhost.exe
TpTrace Verbose: 0 : ###, 5, 2019/04/08, 11:24:34.334,<PHONE_NUMBER>610, vstest.console.dll, Launching default test Host Process C:\Program Files\dotnet\sdk\2.1.503\Extensions\..//TestHost\testhost.exe with arguments  --port ### --endpoint <IP_ADDRESS>:###--role client --parentprocessid #### --diag "########################\bin\Release\diag_TestFindCommand.host.19-04-08_11-24-34_33102_1.txt" --tracelevel 4 --telemetryoptedin false
TpTrace Verbose: 0 : ###, 5, 2019/04/08, 11:24:34.334, ########, vstest.console.dll, DefaultTestHostManager: Starting process 'C:\Program Files\dotnet\sdk\2.1.503\Extensions\..//TestHost\testhost.exe' with command line ' --port ####--endpoint <IP_ADDRESS>:######--role client --parentprocessid ### --diag "########################\bin\Release\diag_TestFindCommand.host.19-04-08_11-24-34_33102_1.txt" --tracelevel 4 --telemetryoptedin false'

@silentnightwalk from the logs the location from where vstest is trying to run the testhost is correct, & from the above exception the only conclusion I can draw is that the testhost.exe is not present at that location.
I'm not sure why that would be, but can you confirm if on your build agents under path C:\Program Files\dotnet\sdk\2.1.503\TestHost, is testhost even present? Or alternatively can you please try to get latest dotnet sdk on your machines?

@mayankbansal018 , thanks!
I'll try to update or confirm the presence of testhost tomorrow. Sorry for misleading info: we have fresh dotnet on other machine, but on this one it's outdated...

Haven't yet had managed to update version. Hopefully, i'll do it on monday.
(i'll delete these two last comments later)

Any update on it?

Sorry for late reply.
After version update, now it's fine.
Not sure what the problem was.

Sure, please comment back here, if you find this issue again, I'll reopen it.

I got the same issue recently, the testhost file is there. I finally fixed it by reinstall VS and add C:\Program Files\dotnet to PATH


Chevron column should be frozen

This might be a bad idea, but I think chevron column should be frozen.

OK the frozen line is really ugly when expanded.

@alump are we doing this?

Yes, now we have nice hiding frozen line, so freeze that column, freeze it good!


Make message receiving not allocate memory on each read

This change was not stable and had little impact on performance. The PR is abandoned.


Ollie: Watch smartslix.com

Ollie requests the watch of the watch_keyword smartslix\.com. See the MS search here and the Stack Exchange search in text, in URLs, and in code.

smartslix\.com has been seen in 1 true positive, 0 false positives, and 0 NAAs.

smartslix\.com has been seen in 1 true positive, 0 false positives, and 0 NAAs.

Rejected by Spevacus in Charcoal HQ. Reason: 'Ninja'd'

Rejected by Spevacus in Charcoal HQ. Reason: 'Ninja'd'


Improve the P/Invoke Layer

:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

README.md
:white_check_mark:Succeeded

binding/Binding.Shared/NativeDelegateContext.cs
:white_check_mark:Succeeded

binding/Binding/GRContext.cs
:white_check_mark:Succeeded

binding/Binding/SKAbstractManagedStream.cs
:white_check_mark:Succeeded

binding/Binding/SKAbstractManagedWStream.cs
:white_check_mark:Succeeded

binding/Binding/SKBitmap.cs
:white_check_mark:Succeeded

binding/Binding/SKCanvas.cs
:white_check_mark:Succeeded

binding/Binding/SKCodec.cs
:white_check_mark:Succeeded

binding/Binding/SKColor.cs
:white_check_mark:Succeeded

binding/Binding/SKColorSpace.cs
:white_check_mark:Succeeded

binding/Binding/SKColorTable.cs
:white_check_mark:Succeeded

binding/Binding/SKImage.cs
:white_check_mark:Succeeded

binding/Binding/SKImageFilter.cs
:white_check_mark:Succeeded

binding/Binding/SKMatrix.cs
:white_check_mark:Succeeded

binding/Binding/SKObject.cs
:white_check_mark:Succeeded

binding/Binding/SKPaint.cs
:white_check_mark:Succeeded

binding/Binding/SKPath.cs
:white_check_mark:Succeeded

binding/Binding/SKPathEffect.cs
:white_check_mark:Succeeded

binding/Binding/SKPicture.cs
:white_check_mark:Succeeded

binding/Binding/SKRegion.cs
:white_check_mark:Succeeded

binding/Binding/SKStream.cs
:white_check_mark:Succeeded

binding/Binding/SKSurface.cs
:white_check_mark:Succeeded

binding/Binding/SKTypeface.cs
:white_check_mark:Succeeded

binding/Binding/SkiaApi.cs
:white_check_mark:Succeeded

binding/Binding/Util.cs
:white_check_mark:Succeeded

externals/skia
:white_check_mark:Succeeded

scripts/pipeline.groovy
:white_check_mark:Succeeded

tests/Tests/SKApiTests.cs
:white_check_mark:Succeeded

tests/Tests/SKMatrixTests.cs
:white_check_mark:Succeeded

tests/Tests/SKPaintTest.cs
:white_check_mark:Succeeded

tests/Tests/SKPathTest.cs
:white_check_mark:Succeeded

tests/Tests/SKTypefaceTest.cs
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


chore: update KB_LEAVE_MEMBER_POD_IP to KB_LEAVE_MEMBER_POD_FQDN

Codecov Report
Attention: Patch coverage is 0% with 8 lines in your changes missing coverage. Please review.

Project coverage is 0.00%. Comparing base (88ea9d0) to head (554cc46).

Files with missing lines
Patch %
Lines

...qdrant/scripts-ut-spec/qdrant_member_leave_spec.sh
0.00%
6 Missing :warning:

...cripts-ut-spec/redis_sentinel_member_leave_spec.sh
0.00%
2 Missing :warning:

Additional details and impacted files
@@          Coverage Diff          @@
##            main   #1314   +/-   ##
=====================================
  Coverage   0.00%   0.00%
=====================================
  Files         69      69
  Lines       6685    6685
=====================================
  Misses      6685    6685

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

fix issues: https://github.com/apecloud/kubeblocks/issues/8642


[Doc][Resource Center] Remind users not to operate resources directly through storage systems

Search before asking

[X] I had searched in the issues and found no similar feature requirement.

Description

When users manage their resource files in resource center, data related to resources is recorded in database. If they delete / rename the resources directly through the storage system (like HDFS, S3) instead of resource center, there will be inconsistency. In that case, they have to modify the database to fix things.
Add notice in docs to remind users not to operate DS related resources directly through the storage system.

Documentation Links
https://dolphinscheduler.apache.org/en-us/docs/dev/user_doc/guide/resource/file-manage.html
Are you willing to submit a PR?

[X] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

The reason is that DS and third-party storage system are two different metadata. At present, there is an optimization process in #10495. This problem will no longer exist after the implementation of this feature is done. @EricGao888

The reason is that DS and third-party storage system are two different metadata. At present, there is an optimization process in #10495. This problem will no longer exist after the implementation of this feature is done. @EricGao888

Cool! Thx for the info @SbloodyS

I will close this issue as soon as #10495 resolved.

I think this issue can be close now, WDYT? @SbloodyS @EricGao888

I think this issue can be close now, WDYT? @SbloodyS @EricGao888

Yes. I'll close it.


Transition to using overpass API for fetching OSM data

Is your feature request related to a problem? Please describe.
Fetching OSM data is time-consuming, requires downloading a ready-made extract (country-wide or city-wide) and uses an external dependency (osm2pgsql tool).
Describe the solution you'd like
Fetch only the tags we need directly with the bounding box from Overpass API (https://wiki.openstreetmap.org/wiki/Overpass_API) using the Python library https://python-overpy.readthedocs.io/en/latest/ . This is most likely faster and simpler, and makes our OSM script similar to our other import scripts.

Done!


Fix error parsing & introduce eip1559

Previous error parsing method was failing for transactions with paymasterAndData - fixed this.
Introduced EIP-1559 (PRO-1459)
Tested against: https://github.com/eth-infinitism/bundler-spec-tests/commit/94a0634ebd6e6da657e863c134d20bc46724ed0f
With EP v0.6.0

@ch4r10t33r it's updated from 0.0.4 to 0.0.5.


inaccessible packages path in context breaks rezolve when lack of package should be what breaks a rezolve

It appears that paths added to the REZ_PACKAGES_PATH var must be available from the point of view of all resolving users, even if no resolved packages in the request are being served from that location;
The presence of an inaccessible path in an environment variable is what breaks rez, not the lack of access to a package, when the condition that should break rez, is the inability to locate a desired package on the contexts present search paths.

(false alarm)


[$500]  IOU - Can't download receipt image

If you haven’t already, check out our contributing guidelines for onboarding and email<EMAIL_ADDRESS>to request to join our Slack channel!

Version Number: <IP_ADDRESS>
Reproducible in staging?:  Y
Reproducible in production?:  Y
If this was caught during regression testing, add the test name, ID and link from TestRail:
Email or phone of affected tester (no customers):
Logs: https://stackoverflow.com/c/expensify/questions/4856
Expensify/Expensify Issue URL:
Issue reported by: Applause - Internal Team
Slack conversation:
Action Performed:

Navigate to https://staging.new.expensify.com/
Navigate to a workspace chat
Go offline
Do a scan IOU
Download the image

Expected Result:
I should be able to download it
Actual Result:
Can't download the receipt image
Workaround:
Unknown
Platforms:

Which of our officially supported platforms is this issue occurring on?

[x] Android: Native
[x] Android: mWeb Chrome
[x] iOS: Native
[x] iOS: mWeb Safari
[ ] MacOS: Chrome / Safari
[ ] MacOS: Desktop

Screenshots/Videos
Add any screenshot/video evidence

https://github.com/Expensify/App/assets/78819774/5d116036-7648-453d-952e-ce43cb0752e8
View all open jobs on GitHub

Proposal
Please re-state the problem that we are trying to solve in this issue.
Can not download receipt when offline
What is the root cause of that problem?
When we offline the image doest not upload to S3 show the source file is local but we have isAuthTokenRequired is true, so the URL has a encryptedSourceUrl. That is wrong
We do not return the isLocalFile here so isAuthTokenRequired is true.
https://github.com/Expensify/App/blob/659f509b7e1093252d7c2ea14d0082ee78dd6503/src/libs/ReceiptUtils.ts#L45-L49
What changes do you think we should make in order to solve the problem?
We shouldupdate
const isLocalFile = path.startsWith('blob:') || path.startsWith('file:');
return {thumbnail: null, image: path, isLocalFile };

What alternative solutions did you explore? (Optional)

@shubham1206agra what do you think of @namhihi237's proposal? Is this a fairly simple change? I think it would be ideal for this to work, but this feels like a fairly unusual flow for someone to follow

@shubham1206agra what do you think of @namhihi237's proposal? Is this a fairly simple change? I think it would be ideal for this to work, but this feels like a fairly unusual flow for someone to follow

Actually, it can be quite common for someone to follow this route. Example is what if I clicked the image of a receipt, and I just want a copy of the image in my local storage too. Then this flow can be followed

@namhihi237's proposal looks good to me.
🎀👀🎀 C+ reviewed

Sent offers via Upwork. @shubham1206agra could you please complete the BZ checklist?

@sakluger accepted

Thanks @namhihi237! I completed your payment.

BugZero Checklist: The PR fixing this issue has been merged! The following checklist (instructions) will need to be completed before the issue can be closed:

[x]  [@shubham1206agra] The PR that introduced the bug has been identified. Link to the PR: https://github.com/Expensify/App/pull/30747
[x]  [@shubham1206agra] The offending PR has been commented on, pointing out the bug it caused and why, so the author and reviewers can learn from the mistake. Link to comment: https://github.com/Expensify/App/pull/30747/files#r1450354884
[x]  [@shubham1206agra] A discussion in #expensify-bugs has been started about whether any other steps should be taken (e.g. updating the PR review checklist) in order to catch this type of bug sooner. Link to discussion: Not required
[x]  [@shubham1206agra] Determine if we should create a regression test for this bug. No as this is an edge case missed by ttps://github.com/Expensify/App/pull/30747
[x]  [@shubham1206agra] If we decide to create a regression test for the bug, please propose the regression test steps to ensure the same bug will not reach production again.
[ ]  [@bfitzexpensify] Link the GH issue for creating/updating the regression test once above steps have been agreed upon:

@sakluger Checklist done

Thank you! All paid out 👍


blazed.company

Blazed Company - Company management hub (virtual office/workspace).
Requirements:

Use Firebase for authentication, cloud functions, and static file hosting.
Use Google Workspaces to create a virtual workspace.
Use Google Docs, Google Sheets, and Google Forms to collect data and to pragmatically create Documents and Spreadsheets

Tasks for Blazed Company

Integration Tasks

[ ] Integrate Firebase { Docs }
[ ] Integrate Google Calendar { Docs }
[ ] Integrate Google Chat { Docs }
[ ] Integrate Google Tasks { Docs }
[ ] Integrate Google Docs { API Docs }
[ ] Integrate Google Sheets { API Docs }
[ ] Integrate Google Forms { API Docs }

Development Tasks

[ ] Develop Basic Prototype
[ ] Create Feed Page
[ ] Create Profile Page

UX Testing Tasks

[ ] Alpha Testing
[ ] Beta Testing
[ ] Stage Testing

Deployment Tasks

[ ] Deploy Production Project


remove appCheckToken for android

I thought of something... will this be a problem when we want to list the app on the play store?

Closing as we have a workaround with entering a phone number.


Allow variations on 'primary' in isShownAt for TN

This is simple enough.  I will merge this if nobody else objects.


Issue 46346: Perf issues rendering HTML wikis

Rationale
We've seen servers get bogged down with wiki rendering.
Changes

Don't double-render the page, once for HTML and once for ClientDependencies. Do it once for both and use our caching mechanism.
Reduce usage of wildly inefficient DOM NodeList implementation method
Don't cache as many wikis - they're potentially huge

I'm not anticipating a 22.9.1 release but there's another fix I think we may push to labkey.org via a snapshot build. https://github.com/LabKey/premium/pull/338


Fix tests

This fixes dependencies for testing and installation, and also updates the tests to run on Python 3.5, as that's the latest stable 3.x release.

Ping on this?


intern dns strings to improve memory usage

What does this PR do?
Uses string interning library to reduce duplicate allocations of strings
Motivation
want to improve memory usage in agent
Describe how to test your changes
Write here in detail how you have tested your changes
and instructions on how this should be tested in QA.
Checklist

[ ] A release note has been added or the changelog/no-changelog label has been applied.
[ ] The need-change/operator and need-change/helm labels has been applied if applicable.
[ ] The appropriate team/.. label has been applied, if known.
[ ] If known, an appropriate milestone has been selected; otherwise the Triage milestone is set.
[ ] The config template has been updated if applicable.

Note: Adding GitHub labels is only possible for contributors with write access.

Would it make sense to either re-use that interner, or convert both to use go4.org/intern?

I took a look at the dogstatsd interner, and I don't think it suits this use case because it doesn't allow for garbage collection. It simply resets the entire map when a maximum size is hit. I also don't think it is thread-safe.

@db Can you please include the data/graphs about the performance gains from this PR. Thanks!

I took a look at the dogstatsd interner, and I don't think it suits this use case because it doesn't allow for garbage collection. It simply resets the entire map when a maximum size is hit. I also don't think it is thread-safe.

Yeah, that makes sense.  I'm working on a GC-ing intern tool here but it's not ready yet.

I'm working on a GC-ing intern tool here but it's not ready yet.

FWIW go4.org/intern is effectively written by the Go core team. It is part of a larger effort to build a new IP address type. https://tailscale.com/blog/netaddr-new-ip-type-for-go/

Yeah, I had a look -- it lacks some things that I need for this usecase (in particular, exposing hashes).  Thanks!

@derekwbrown looks like pkg/network/dns/snooper_test.go needs to be updated.


Add support for async db calls

Can one of the admins verify this patch?  Type 'ok to test' to test.

So beyond the specific comments;
in general I'm afraid this is a good brush, a great brush, applied too broadly. It's generally speaking a bad idea to force every single plugin using Namelayer to be using the "async" versions of the DB calls; you don't know for sure what they are doing, and a much better technique is to leave the API behavior alone for existing methods and add new wrappers that are async, that Namelayer can use internally (and we can test and assert that its safe to do so).
This leads to the final critique: the naming convention is backwards; methods marked "async" are actually the synchronous versions, whereas the version w/o "async" in the name always spawn an asynch thread. Bit confusing.
Otherwise good effort, ping me when you've had a chance to react to / address some of the issues I've raised.

@ProgrammerDan
To address all of your mercury notes, I consolidated the check into the message method: https://github.com/Civcraft/NameLayer/pull/208/files#diff-bd66cb6ca613984f83e554cd4a94dfeeR12
The reason i removed most(if not all) of the invalidations, is because the invalidations tells all other servers to recache that group. It seems like most, if not all changes to groups caused a recache message, which would result in a (dbcall * number-of-servers). If we can expect all other servers to reliably receive the recache message, then we can also expect it to reliably receive the messages to just apply whatever changes happened to its own cache. To support this I went around to each part that caused a dbwrite and paired with it a mercury call(obv if available).
Example here:
https://github.com/Civcraft/NameLayer/pull/208/files#diff-c69a82c6840d6af9abc1feda6fc3086fR161
New mercury message handling here:
https://github.com/Civcraft/NameLayer/pull/208/files#diff-1f88aaa5d64cab1c0f1a4bed8eff54d8R47
So with the above changes, when mercury gets a merge message, it will call the same method that would be called locally, the only difference is the 'savetodb' boolean that will disable the db call and mercury message for that call.
Yea I can see what you mean about the async naming being confusing. For the functionality of it, perhaps making a config option for the async db calls to kick in would be ideal?

Ideally you would do as I described; keep the published API behavior unchanged(synchronous) but internally alter all NameLayer owned calls to use the asynchronous method.

@suirad Any chance of altering default as described? Again, looks like you did a great job in this pull, but I'm not convinced it's a good idea to force everything that leverages NameLayer to use async DB calls at this point.
Instead, use the async inside namelayer exclusively but leave the previously exposed methods behaving equivalent to prior (sync) to prevent regressions.
I'll correct it in this PR tonight most likely if I haven't heard back from you by then.

That's essentially what was changed. The published api is the same, so I didn't modify any call sites; but internally each write pushes the db call off to an async task.

That, specifically, is what I have an issue with. Previously plugins could be sure that the actual DB action was synchronous; at the return of the method call, the DB insert / update was done as well.
Now, that isn't the case.
This is not a good practice.
For external plugins to continue leveraging this synchronous behavior, they now have to call the method name + confusingly "async".
It's just backwards. Easy fix. Code is good, method names are bad. Don't swap out synchronous behavior for asynchronous behavior in publicly accessible methods on longstanding code-bases where many plugins leverage those methods, you will break things.
It's fine to convert all internal uses to async, but the new method name should be the async behavior, not the old method name.

Ah okay that makes sense. I shall adjust it when I get the time. Will probably be tomorrow night.


Re-encode dictionaries in selection kernels

Which issue does this PR close?

Closes #506
Relates to #2832
Rationale for this change

MutableArrayData blindly concatenates the dictionary values arrays together when given multiple different dictionaries. This is straightforward and often fast, but degrades spectacularly if either of the following hold:

The dictionary values are large
Not all dictionary values are referenced by the keys (e.g. the array has been filtered / sliced, read from parquet)

This leads to high memory usage, wasted CPU cycles, and potential byte array offset overflow
What changes are included in this PR?

Are there any user-facing changes?

I think this is now ready for review, benchmarks suggest re-encoding dictionaries is on the order of ~10µs for 1024 rows.
The difficulty is the old method is very fast most of the time, on the order of 1µs, and so always re-encoding would result in a significant performance regression in the common case. To avoid this, I've stuck a fairly pessimistic heuristic that will only re-encode if the concatenation of the dictionary values would end up with more elements than the output has keys.
This helps to avoid the worst case most of the time, and I think is an acceptable trade-off but welcome other thoughts.
Interleave with the same dictionary (which won't merge)
interleave dict(20, 0.0) 100 [0..100, 100..230, 450..1000]
                        time:   [1.6196 µs 1.6204 µs 1.6212 µs]
Found 3 outliers among 100 measurements (3.00%)
  2 (2.00%) high mild
  1 (1.00%) high severe

interleave dict(20, 0.0) 400 [0..100, 100..230, 450..1000]
                        time:   [4.4878 µs 4.4911 µs 4.4946 µs]
Found 4 outliers among 100 measurements (4.00%)
  3 (3.00%) high mild
  1 (1.00%) high severe

interleave dict(20, 0.0) 1024 [0..100, 100..230, 450..1000]
                        time:   [10.931 µs 10.937 µs 10.944 µs]
Found 6 outliers among 100 measurements (6.00%)
  3 (3.00%) high mild
  3 (3.00%) high severe

interleave dict(20, 0.0) 1024 [0..100, 100..230, 450..1000, 0..1000]
                        time:   [10.675 µs 10.678 µs 10.681 µs]
Found 6 outliers among 100 measurements (6.00%)
  4 (4.00%) high mild
  2 (2.00%) high severe

interleave dict_sparse(20, 0.0) 100 [0..100, 100..230, 450..1000]
                        time:   [1.6390 µs 1.6399 µs 1.6407 µs]
Found 6 outliers among 100 measurements (6.00%)
  3 (3.00%) low mild
  2 (2.00%) high mild
  1 (1.00%) high severe

interleave dict_sparse(20, 0.0) 400 [0..100, 100..230, 450..1000]
                        time:   [4.5065 µs 4.5080 µs 4.5096 µs]
Found 3 outliers among 100 measurements (3.00%)
  2 (2.00%) high mild
  1 (1.00%) high severe

interleave dict_sparse(20, 0.0) 1024 [0..100, 100..230, 450..1000]
                        time:   [11.008 µs 11.014 µs 11.020 µs]
Found 3 outliers among 100 measurements (3.00%)
  2 (2.00%) high mild
  1 (1.00%) high severe

interleave dict_sparse(20, 0.0) 1024 [0..100, 100..230, 450..1000, 0..1000]
                        time:   [10.684 µs 10.688 µs 10.692 µs]
Found 6 outliers among 100 measurements (6.00%)
  1 (1.00%) low mild
  4 (4.00%) high mild
  1 (1.00%) high severe

Interleave with different dictionaries where one is sparse (will merge)
interleave dict_distinct 100
                        time:   [5.2581 µs 5.2605 µs 5.2631 µs]
Found 6 outliers among 100 measurements (6.00%)
  2 (2.00%) high mild
  4 (4.00%) high severe

interleave dict_distinct 1024
                        time:   [5.2225 µs 5.2256 µs 5.2289 µs]
Found 8 outliers among 100 measurements (8.00%)
  5 (5.00%) high mild
  3 (3.00%) high severe

interleave dict_distinct 2048
                        time:   [5.0365 µs 5.0419 µs 5.0478 µs]
Found 2 outliers among 100 measurements (2.00%)
  1 (1.00%) high mild
  1 (1.00%) high severe

The concatenate kernel is typically much faster, and so merging does represent a non-trivial performance regression. (Note this is concatenating 2 arrays of 1024, whereas the interleave benchmarks are interleaving to produce an array of 1024 rows).
concat str_dict 1024    time:   [2.7670 µs 2.7692 µs 2.7711 µs]

concat str_dict_sparse 1024
                        time:   [21.122 µs 21.136 µs 21.151 µs]
Found 1 outliers among 100 measurements (1.00%)
  1 (1.00%) high severe

Converting to draft whilst I continue to think about how we should be handling this, especially in light of the RunEncodedArray work

Closing this for now, may re-open in future if I have further ideas in this space

The latest benchmarks are
concat str_dict 1024    time:   [4.1847 µs 4.1892 µs 4.1935 µs]
                        change: [-11.560% -11.248% -10.897%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 3 outliers among 100 measurements (3.00%)
  2 (2.00%) high mild
  1 (1.00%) high severe

concat str_dict_sparse 1024
                        time:   [11.443 µs 11.452 µs 11.463 µs]
                        change: [+13.983% +14.341% +14.645%] (p = 0.00 < 0.05)
                        Performance has regressed.

interleave dict(20, 0.0) 100 [0..100, 100..230, 450..1000]
                        time:   [3.4982 µs 3.5000 µs 3.5020 µs]
                        change: [+27.355% +27.663% +28.183%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 2 outliers among 100 measurements (2.00%)
  1 (1.00%) high mild
  1 (1.00%) high severe

interleave dict(20, 0.0) 400 [0..100, 100..230, 450..1000]
                        time:   [6.5197 µs 6.5226 µs 6.5257 µs]
                        change: [+6.0766% +6.3786% +6.6982%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 3 outliers among 100 measurements (3.00%)
  1 (1.00%) high mild
  2 (2.00%) high severe

interleave dict(20, 0.0) 1024 [0..100, 100..230, 450..1000]
                        time:   [12.783 µs 12.787 µs 12.791 µs]
                        change: [-4.4718% -4.2584% -4.1254%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 4 outliers among 100 measurements (4.00%)
  1 (1.00%) low mild
  1 (1.00%) high mild
  2 (2.00%) high severe

interleave dict(20, 0.0) 1024 [0..100, 100..230, 450..1000, 0..1000]
                        time:   [13.352 µs 13.358 µs 13.364 µs]
                        change: [-2.1913% -1.9010% -1.6320%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 8 outliers among 100 measurements (8.00%)
  4 (4.00%) high mild
  4 (4.00%) high severe

interleave dict_sparse(20, 0.0) 100 [0..100, 100..230, 450..1000]
                        time:   [3.5297 µs 3.5310 µs 3.5323 µs]
                        change: [+26.230% +26.553% +26.772%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 7 outliers among 100 measurements (7.00%)
  5 (5.00%) high mild
  2 (2.00%) high severe

interleave dict_sparse(20, 0.0) 400 [0..100, 100..230, 450..1000]
                        time:   [6.5106 µs 6.5152 µs 6.5201 µs]
                        change: [+6.5054% +6.8053% +7.1029%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 3 outliers among 100 measurements (3.00%)
  3 (3.00%) high severe

interleave dict_sparse(20, 0.0) 1024 [0..100, 100..230, 450..1000]
                        time:   [13.007 µs 13.010 µs 13.014 µs]
                        change: [-3.6668% -3.2717% -2.9611%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 7 outliers among 100 measurements (7.00%)
  4 (4.00%) high mild
  3 (3.00%) high severe

interleave dict_sparse(20, 0.0) 1024 [0..100, 100..230, 450..1000, 0..1000]
                        time:   [13.288 µs 13.297 µs 13.307 µs]
                        change: [-2.8566% -2.6176% -2.4714%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 4 outliers among 100 measurements (4.00%)
  3 (3.00%) high mild
  1 (1.00%) high severe

interleave dict_distinct 100
                        time:   [5.5472 µs 5.5536 µs 5.5606 µs]
                        change: [+0.6246% +0.9056% +1.1312%] (p = 0.00 < 0.05)
                        Change within noise threshold.
Found 5 outliers among 100 measurements (5.00%)
  4 (4.00%) high mild
  1 (1.00%) high severe

interleave dict_distinct 1024
                        time:   [5.5718 µs 5.5794 µs 5.5875 µs]
                        change: [+0.9659% +1.2001% +1.4061%] (p = 0.00 < 0.05)
                        Change within noise threshold.
Found 10 outliers among 100 measurements (10.00%)
  6 (6.00%) high mild
  4 (4.00%) high severe

interleave dict_distinct 2048
                        time:   [5.4061 µs 5.4136 µs 5.4212 µs]
                        change: [-2.1558% -1.8430% -1.5765%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 3 outliers among 100 measurements (3.00%)

My takeaway from this is that the performance regression is not significant enough in absolute terms to warrant not making this the default behaviour.

Looking at a flamegraph for the regressed interleave dict(20, 0.0) 100 [0..100, 100..230, 450..1000] shows a non-trivial amount of time spent in should_merge_dictionary_values associated with the ArrayData conversion. I may see if I can optimise this further

With the changes in https://github.com/apache/arrow-rs/pull/3558/commits/c2d40092cbf41f8cf0838b2239ec8392017da6d3 there is no longer a performance regression, except when re-encoding
interleave dict(20, 0.0) 100 [0..100, 100..230, 450..1000]
                        time:   [2.7514 µs 2.7549 µs 2.7582 µs]
                        change: [+0.2105% +0.4287% +0.7010%] (p = 0.00 < 0.05)
                        Change within noise threshold.
Found 3 outliers among 100 measurements (3.00%)
  1 (1.00%) low mild
  1 (1.00%) high mild
  1 (1.00%) high severe

interleave dict(20, 0.0) 400 [0..100, 100..230, 450..1000]
                        time:   [5.7040 µs 5.7069 µs 5.7101 µs]
                        change: [-7.1155% -6.8505% -6.5598%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 4 outliers among 100 measurements (4.00%)
  2 (2.00%) high mild
  2 (2.00%) high severe

interleave dict(20, 0.0) 1024 [0..100, 100..230, 450..1000]
                        time:   [11.952 µs 11.958 µs 11.964 µs]
                        change: [-10.702% -10.503% -10.379%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 5 outliers among 100 measurements (5.00%)
  4 (4.00%) high mild
  1 (1.00%) high severe

interleave dict(20, 0.0) 1024 [0..100, 100..230, 450..1000, 0..1000]
                        time:   [12.001 µs 12.006 µs 12.011 µs]
                        change: [-12.141% -11.932% -11.799%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 7 outliers among 100 measurements (7.00%)
  6 (6.00%) high mild
  1 (1.00%) high severe

interleave dict_sparse(20, 0.0) 100 [0..100, 100..230, 450..1000]
                        time:   [2.7432 µs 2.7468 µs 2.7506 µs]
                        change: [-1.5949% -1.2917% -0.9539%] (p = 0.00 < 0.05)
                        Change within noise threshold.
Found 1 outliers among 100 measurements (1.00%)
  1 (1.00%) high severe

interleave dict_sparse(20, 0.0) 400 [0..100, 100..230, 450..1000]
                        time:   [5.6984 µs 5.7024 µs 5.7064 µs]
                        change: [-6.7690% -6.4981% -6.2229%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 5 outliers among 100 measurements (5.00%)
  4 (4.00%) high mild
  1 (1.00%) high severe

interleave dict_sparse(20, 0.0) 1024 [0..100, 100..230, 450..1000]
                        time:   [12.065 µs 12.071 µs 12.077 µs]
                        change: [-10.666% -10.329% -10.110%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 5 outliers among 100 measurements (5.00%)
  1 (1.00%) high mild
  4 (4.00%) high severe

interleave dict_sparse(20, 0.0) 1024 [0..100, 100..230, 450..1000, 0..1000]
                        time:   [11.995 µs 12.000 µs 12.007 µs]
                        change: [-12.334% -12.126% -11.999%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 8 outliers among 100 measurements (8.00%)
  1 (1.00%) low mild
  5 (5.00%) high mild
  2 (2.00%) high severe

interleave dict_distinct 100
                        time:   [4.9384 µs 4.9423 µs 4.9467 µs]
                        change: [-10.644% -10.405% -10.218%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 5 outliers among 100 measurements (5.00%)
  4 (4.00%) high mild
  1 (1.00%) high severe

interleave dict_distinct 1024
                        time:   [4.8532 µs 4.8597 µs 4.8655 µs]
                        change: [-12.302% -12.073% -11.862%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 2 outliers among 100 measurements (2.00%)
  1 (1.00%) high mild
  1 (1.00%) high severe

interleave dict_distinct 2048
                        time:   [4.8942 µs 4.9006 µs 4.9081 µs]
                        change: [-11.126% -10.785% -10.446%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 3 outliers among 100 measurements (3.00%)
  1 (1.00%) high mild
  2 (2.00%) high severe

🎉


To do list

remove old class DbcParser: move the Message and Signal class into Model folder
add more regression tests for the Packer methods

done.


Disable login button on submit

Disable login button on submit (and/or maybe add some sort of loading animation) to avoid possible errors by the user double-clicking the button instead of single-clicking.
See Sentry error PROFILES-5D for an example. An integrity constraint violation occurred because it tried to add the same user twice to the DB.

Also issue PROFILES-63

Illuminate\Database\QueryException
SQLSTATE[23000]: Integrity constraint violation: Duplicate entry for key 'role_user.PRIMARY' (SQL: insert into role_user (role_id, user_id) values (X, XXXX))


skip: bind address flag, serve skip.pac via http

Add --bind address for local address. Uses kingpin to parse flags simply
because this library is already in use in this repo.
The skip.pac file is now interpreted as a template so that the local
address can be filled in and served on http (instead of previously just
installing it with a file://-URL).
For this, we use the new "go:embed" feature to embed the template file
into the binary (hence the update to go-1.16).
Configuring the skip.pac file with an http://-URL appears to make this
work in chrome :tada:.
Also:

update to go-1.16 (for go:embed, see above), update/cleanup related do docs
update golangci-lint version in Makefile
go mod tidy

This change is

skip/main.go, line 68 at r1 (raw file):

	}
	mux := http.NewServeMux()
	mux.Handle("localhost/skip.pac", http.HandlerFunc(handleWPAD))

Is there a reason why we don't just register skip.pac at the root like
mux.Handle("/skip.pac", http.HandlerFunc(handleWPAD))

localhost doesn't work well in my setup, where I run skip in a local VM. But with skip.pac at the root it worked fine so far.

skip/main.go, line 68 at r1 (raw file):
Previously, matzf (Matthias Frei) wrote…
Yes, otherwise this would make impossible to proxy any request matching /skip.pac. This would unlikely be an issue, but it feels just wrong to ignore it. It's a good point though, I forgot to revisit this when adding the --bind option. I've now added another route for the local bind IP address. Does that work for you?

Works fine now. Thanks! I was lazy and used <IP_ADDRESS> as the local bind address and then it obviously still doesn't work. But binding to a specific address seems to be cleaner anyway and I think this is a good solution now.


Improve MultilabelAveragePrecisionEvaluator

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 100.00%. Comparing base (85c3f5e) to head (073a528).
Report is 6 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff            @@
##              main      #333   +/-   ##
=========================================
  Coverage   100.00%   100.00%
=========================================
  Files          100       100
  Lines         2428      2429    +1
  Branches       197       196    -1
=========================================
+ Hits          2428      2429    +1

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.