process udf function fail with batch but works with stream [solved]

Hello.
I wrote udf function and it's fails with batch in .tick file.  Can you help me with these issue?
kapacitor.log:

[httpd] ::1 - - [08/Dec/2016:15:31:14 +0000] "PATCH /kapacitor/v1/tasks/forecast_disk_capacity
HTTP/1.1" 200 719 "-" "KapacitorClient" 5ac6afaf-bd5b-11e6-8034-000000000
000 17726
[task_master:main] 2016/12/08 15:31:14 D! Starting task: forecast_disk_capacity
[task_master:main] 2016/12/08 15:31:14 I! Started task: forecast_disk_capacity
[task_master:main] 2016/12/08 15:31:14 D! digraph forecast_disk_capacity {
query1 -> Forecast2;
Forecast2 -> alert3;
alert3 -> log4;
}
[httpd] ::1 - - [08/Dec/2016:15:31:14 +0000] "PATCH /kapacitor/v1/tasks/forecast_disk_capacity HTTP/1.1" 200 834 "-" "KapacitorClient" 5ac972b5-bd5b-11e6-8035-000000000000 13147
[httpd] ::1 - - [08/Dec/2016:15:31:14 +0000] "GET /kapacitor/v1/tasks?dot-view=attributes&fields=link&limit=100&offset=0&pattern=forecast_disk_capacity&replay-id=&script-format=formatted HTTP/1.1" 200 43 "-" "KapacitorClient" 5acdfb00-bd5b-11e6-8036-000000000000 521
[forecast_disk_capacity:Forecast2] 2016/12/08 15:31:14 I!P Starting agent for Forecast
[influxdb] 2016/12/08 15:31:16 D! linking subscriptions for cluster localhost
[forecast_disk_capacity:query1] 2016/12/08 15:32:14 D! starting next batch query: SELECT min(mount_point_free_kb) / 1024 / 1024 FROM monitoring."default".disk_capacity WHERE host = 'dwhfrn03' AND time >= '2016-12-03T15:32:14.433781029Z' AND time < '2016-12-08T15:32:14.433781029Z' GROUP BY time(1h, 0s), mount_point
[forecast_disk_capacity:Forecast2] 2016/12/08 15:32:14 I!P mount_point_free_kb
[forecast_disk_capacity:Forecast2] 2016/12/08 15:32:14 I!P Traceback (most recent call last):
[forecast_disk_capacity:Forecast2] 2016/12/08 15:32:14 I!P   File "/usr/lib/python2.7/dist-packages/agent.py", line 122, in _read_loop
[forecast_disk_capacity:Forecast2] 2016/12/08 15:32:14 I!P     self.handler.point(request.point)
[forecast_disk_capacity:Forecast2] 2016/12/08 15:32:14 I!P   File "/kapacitor_rules/udf/forecast.py", line 117, in point
[forecast_disk_capacity:Forecast2] 2016/12/08 15:32:14 I!P     value = point.fieldsDouble[self._field]
[forecast_disk_capacity:Forecast2] 2016/12/08 15:32:14 I!P   File "/usr/local/lib/python2.7/dist-packages/google/protobuf/internal/containers.py", line 462, in getitem
[forecast_disk_capacity:Forecast2] 2016/12/08 15:32:14 I!P     val = self._value_checker.DefaultValue()
[forecast_disk_capacity:Forecast2] 2016/12/08 15:32:14 I!P AttributeError: 'TypeChecker' object has no attribute 'DefaultValue'
[forecast_disk_capacity:Forecast2] 2016/12/08 15:32:14 I!P 2016-12-08 15:32:14,463 ERROR:root: error processing request of type point: 'TypeChecker' object has no attribute 'DefaultValue'
[forecast_disk_capacity:Forecast2] 2016/12/08 15:32:14 I!P Agent finished
[forecast_disk_capacity:Forecast2] 2016/12/08 15:32:14 E! error processing request of type point: 'TypeChecker' object has no attribute 'DefaultValue'
[edge:forecast_disk_capacity|Forecast2->alert3] 2016/12/08 15:32:14 D! closing c: 0 e: 0
[edge:forecast_disk_capacity|query1->Forecast2] 2016/12/08 15:32:14 I! aborting c: 13 e: 10
[forecast_disk_capacity:Forecast2] 2016/12/08 15:32:14 E! error processing request of type point: 'TypeChecker' object has no attribute 'DefaultValue'
[edge:forecast_disk_capacity|alert3->log4] 2016/12/08 15:32:14 D! closing c: 0 e: 0
[task_store] 2016/12/08 15:32:14 D! task forecast_disk_capacity finished
[edge:forecast_disk_capacity|batch->batch0] 2016/12/08 15:32:14 D! closing c: 13 e: 13
[edge:forecast_disk_capacity|query1->Forecast2] 2016/12/08 15:32:14 D! closing c: 13 e: 10
[task_master:main] 2016/12/08 15:32:14 E! Stopped task: forecast_disk_capacity Forecast2: error processing request of type point: 'TypeChecker' object has no attribute 'DefaultValue'
[task_store] 2016/12/08 15:32:14 E! task forecast_disk_capacity finished with error: Forecast2: error processing request of type point: 'TypeChecker' object has no attribute 'DefaultValue'

code of function:
from agent import Agent, Handler
#from scipy import stats
#import math
import numpy
import udf_pb2
import sys
import logging
logging.basicConfig(level=logging.DEBUG, format='%(asctime)s %(levelname)s:%(name)s: %(message)s')
logger = logging.getLogger()

class Forecast(Handler):

    class state(object):
        def __init__(self):
            self._entries = []

        def reset(self):
            self._entries = []

        #def update(self, value, point):
        #    self._entries.append((value, point))
        def update(self, value):
            # tranform to gigabytes
            self._entries.append((value/1024/1024))

        def calc(self):
            print >> sys.stderr, self._entries
            y = numpy.array(self._entries)
            X = numpy.array([ numpy.ones(len(y)), list(range(1,len(y)+1)) ]).transpose()
            #print(X)
            #thetaMulti = ((X'*X)^-1)*X'*y
            X_trans = X.transpose()

            thetaMulti = numpy.linalg.matrix_power(X_trans.dot(X),-1).dot(X_trans).dot(y)

            temp = numpy.power(X_trans.dot(X),-1).dot(X_trans).dot(y)

            iter_count = -thetaMulti[0]/thetaMulti[1] - len(y);

            return iter_count

        #def length(self):
        #    return len(self._entries)

    def __init__(self, agent):
        self._agent = agent

        self._field = None

        self._begin_response = None

        self._state = Forecast.state()

        """
        Keep a rolling window of historically normal data
        When a new window arrives use a two-sided t-test to determine
        if the new window is statistically significantly different.
        """
    def info(self):
        """
        Respond with which type of edges we want/provide and any options we have.
        """
        response = udf_pb2.Response()
        # We will consume batch edges aka windows of data.
        response.info.wants = udf_pb2.BATCH
        # We will produce single points of data aka stream.
        response.info.provides = udf_pb2.STREAM

        # Here we can define options for the UDF.
        # Define which field we should process
        response.info.options['field'].valueTypes.append(udf_pb2.STRING)

        logger.info("info")
        return response

    def init(self, init_req):
        """
        Given a list of options initialize this instance of the handler
        """
        success = True
        msg = ''
        size = 0
        for opt in init_req.options:
            if opt.name == 'field':
                self._field = opt.values[0].stringValue

        if self._field is None:
            success = False
            msg += ' must supply a field name'

        response = udf_pb2.Response()
        response.init.success = success
        response.init.error = msg[1:]
        return response

    def snapshot(self):
        response = udf_pb2.Response()
        response.snapshot.snapshot = ''
        return response

    def restore(self, restore_req):
        response = udf_pb2.Response()
        response.restore.success = False
        response.restore.error = 'not implemented'
        return response

    def begin_batch(self, begin_req):
        #print >> sys.stderr, "START!!!!"
        self._state.reset()
        # Keep copy of begin_batch
        response = udf_pb2.Response()
        response.begin.CopyFrom(begin_req)
        self._begin_response = response

    def point(self, point):
        print >> sys.stderr, self._field
        value = point.fieldsDouble[self._field]
        #self._state.update(value, point)
        self._state.update(value)

    def end_batch(self, end_req):
        interval_count = self._state.calc()
        #msg = "LENGTH "
        #msg += str(length)
        #print >> sys.stderr, msg
        response = udf_pb2.Response()
        response = udf_pb2.Response()
        response.point.time = end_req.tmax
        response.point.name = end_req.name
        response.point.group = end_req.group
        response.point.tags.update(end_req.tags)
        response.point.fieldsDouble["interval_value"] = interval_count
        self._agent.write_response(response)

if __name__ == '__main__':
    # Create an agent
    agent = Agent()

    # Create a handler and pass it an agent so it can write points
    h = Forecast(agent)

    # Set the handler on the agent
    agent.handler = h

    # Anything printed to STDERR from a UDF process gets captured into the Kapacitor logs.
    print >> sys.stderr, "Starting agent for Forecast"
    agent.start()
    agent.wait()
    print >> sys.stderr, "Agent finished"

tick file:
var data = batch
     |query('''
         SELECT min("mount_point_free_kb")/1024/1024
         FROM "monitoring"."default".disk_capacity
         WHERE "host" = 'dwhfrn03'
     ''')
        .period(5d)
        .every(1m)
        .groupBy( time(1h), 'mount_point')

data
    //Run our tTest UDF on the hotend temperature
    @Forecast()
    .field('mount_point_free_kb')
    |alert()
        .crit(lambda: "interval_value" > 0)
        .log('/var/log/kapacitor/alerts_free_disk_space.log')
    |log()

function in kapacitor.conf:
[udf]
  [udf.functions]
    [udf.functions.tTest]
        # Run python
        prog = "/usr/bin/python2"
        # Pass args to python
        # -u for unbuffered STDIN and STDOUT
        # and the path to the script
        args = ["-u", "/kapacitor_rules/udf/ttest.py"]
        # If the python process is unresponsive for 10s kill it
        timeout = "10s"
        # Define env vars for the process, in this case the PYTHONPATH
        [udf.functions.tTest.env]
            PYTHONPATH = "/kapacitor_rules/udf/agent/py"
    [udf.functions.Forecast]
        # Run python
        prog = "/usr/bin/python2"
        # Pass args to python
        # -u for unbuffered STDIN and STDOUT
        # and the path to the script
        args = ["-u", "/kapacitor_rules/udf/forecast.py"]
        # If the python process is unresponsive for 10s kill it
        timeout = "10s"
        # Define env vars for the process, in this case the PYTHONPATH
        [udf.functions.Forecast.env]
            PYTHONPATH = "/kapacitor_rules/udf/agent/py"

The same function works with stream tick:
var data = stream
    |from()
    .measurement('disk_capacity')
    .groupBy('mount_point')
    .where(lambda: "host" == 'dwhfrn03')
    .groupBy('mount_point')
    |window()
        .period(1m)
        .every(1m)

data
    //Run our tTest UDF on the hotend temperature
    @Forecast()
    .field('mount_point_free_kb')
    |alert()
        .crit(lambda: "interval_value" > 0)
        .log('/var/log/kapacitor/alerts_free_disk_space.log')
    |log()

There is difference from stream and batch. When I start my batch I get this message in ponit:
time:<PHONE_NUMBER>000000000
group: "mount_point=/var/adm/ras/livedump"
tags {
   key: "mount_point"
   value: "/var/adm/ras/livedump"
 }
 fieldsDouble {
   key: "minimum"
   value: 0.247661590576
 }

but from tick script waiting 'mount_point_free_kb' tag inside. I just simply fix batch tick:
var data = batch
     |query('''
         SELECT min("mount_point_free_kb")/1024/1024 as minimum
         FROM "monitoring"."default".disk_capacity
         WHERE "host" = 'dwhfrn03'
     ''')
        .period(5d)
        .every(1m)
        .groupBy( time(1h), 'mount_point')

data
    //Run our tTest UDF on the hotend temperature
    @Forecast()
    .field('minimum')
    |alert()
        .crit(lambda: "interval_value" > 0)
        .log('/var/log/kapacitor/alerts_free_disk_space.log')
    |log(

Thank you.


Normalize and update IT docker setups

Many integration tests have docker setups which use old images. In addition the setups are complicated. See if the setup can be consolidated, perhaps in the project pom, and the images updated.
Come up with a process for updating these integration test images as new images are released.

To expand a bit, some projects use older releases of various Docker images, while other projects will need new image published in order to support the migration to Java 11 (#270)

Integration tests should run on images with the same version as the current version of the project.

even if we do nothing else, we can specify the image urls for the various docker images we use as properties in the main pom, and have ITs for the various projects use these property values. This would go a long way to making sure that all ITs are run against the same, most recent images.

pass-messaging-support has some references to docker images that are not used anywhere in the project. I suspect they were copy + pasted from somewhere else. Should remove to avoid confusion

The two remaining code repos with failing ITs will not be updated yet, pending the replacement of Fedora, because of tight coupling with Fedora https://github.com/eclipse-pass/main/issues/302


Intellisesne inside lambda functions (map)

Bicep version
Bicep CLI version 0.27.1 (4b41cb6d4b)
Describe the bug
Intellisense is lost inside map() example. It applies to other lambda functions as well.
To Reproduce

var subnets = []

resource vnet 'Microsoft.Network/virtualNetworks@2023-11-01' = {
  name: 'vnet'
  properties: {
    subnets: map(subnets, subnet => {

    })
  }
}

Works if I use for loop:

It also should work if you have nested map() no matter how many times is nested:

var subnets = []

resource vnet 'Microsoft.Network/virtualNetworks@2023-11-01' = {
  name: 'vnet'
  properties: {
    subnets: map(subnets, subnet => {
      name: subnet.name
      properties: {
        addressPrefix: subnet.addressPrefix
        serviceEndpoints: map(subnet.serviceEndpoints, serviceEndpoint => {
            locations: serviceEndpoint.locations
            //service: serviceEndpoint.service

          })
      }
    })
  }
}

//cc @anthony-c-martin
Additional context
Add any other context about the problem here.

Another case for testing:

var subnets = []

resource vnet 'Microsoft.Network/virtualNetworks@2023-11-01' = {
  name: 'vnet'
  properties: {
    subnets: empty(subnets) ? null : map(subnets, subnet => {

    })
  }
}


feature: Support to deploy a workload packaged as a Helm chart to a workload execution cluster

Feature Description
Some workloads are packaged as helm charts. We need to be able to deploy such applications to a workload execution cluster via KubeStellar.
Proposed Solution
To be able to use a helm chart to deploy a workload to a workload execution cluster via KubeStellar
Want to contribute?

[ ] I would like to work on this issue.

Additional Context
No response

The real work here presumably is instantiating the Helm chart in a WDS, since KubeStellar will get it from the WDS to the WEC.

I think this works ok.  I have done it a few times.  Yes, we are missing some automation around special types/kinds that are required, but that is cared for in another issue.  Closing this for now.


drivers: i2c_ll_stm32_v1: Wait until not busy before disabling i2c

There are a number of bug reports and PRs about I2C bus hangs. The problem I had looked a lot like bug #22751 on the oscilloscope, but neither STOP nor BUSY flags are set before sending START as described in that bug report and in PR #23663. This might be because the problem I had is different, or because the I2C driver is different nowadays compared to 1.14 (it now disables/enables the I2C peripheral for each transaction, for example).
Waiting until the busy flag is cleared before disabling the I2C peripheral helps with an I2C bus hang that has been observed on STM32F429. The I2C bus would stop working after a while when reading from a particular I2C device. After that, the STM32 would only send a start condition immediately followed by a stop condition, until it was reset.
Tested on v2.2.0 on STM32F429, I2C master mode, with and without CONFIG_I2C_STM32_INTERRUPT.

Hey @jonasn-ge  could you check if PR #27291 works for you ? Because when I made that patch just waiting for BUSY to go away did not work, it might be that the master branch version reacts different, I will try your patch in the weekend.
Can your setup deal with disruption of the SDA signal, the 1.14 (and master) driver would hang when disrupting the SDA signal for example by shortly connecting it to GND.  That is the other commit in #27291 that deals with that, giving up after a timeout and reseting the hardware.

Hi @lowlander, yes PR #27291 works. Details:
On vanilla 2.2.0:
I2C sometimes spontaneously stops working.
If I tie SDA to ground for a bit, it causes I2C to stop working either with SCL stuck low or generating clock pulses forever.
With my PR:
I2C doesn't spontaneously stop working.

PR #27291:
The problem of I2C spontaneously locking up seems to be gone (run for 15 minutes so far). I can see the STOP flag detection log from time to time, and clearing STOP does the trick.
If I tie SDA to ground, the write timeout kicks in and recovers the situation. Nice!
It is a bit curious to me that it works, because I had been looking a lot for that stuck STOP flag before finding out that waiting for BUSY to clear helps (at least for me). What caught me was that LL_I2C_IsActiveFlag_STOP() isn't for that STOP flag, so I was looking at the wrong bit.

@jonasn-ge On my system I poll a I2C sensor only every 100 ms or so, and on the old driver it was still "busy" on the start of the next I2C transfer, that's how I found out I could not "just wait" for it to go away.  Somehow the STM I2C hardware is a bit weird :-/

Looks like PR #27291 solves the problem and is close to getting merged. Closing.


Replace master with main

Replace references to master with main in ci.yml
Description
The "master" branch is to be renamed to "main". This will not automatically update any GitHub Action workflows. Hence updating ci.yml to replace references to master with main
Checklist:

[ NA] I have tested my changes. No regression in existing tests.
[ NA] My code is Linted.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Replace references to master with main in ci.yml

Replace references to master with main in ci.yml

We need to revert this. This repo doesn't use main: https://github.com/awslabs/git-secrets/


Simultaneous tests accessing same files give execution error

Proposed solution to join tests.
Ugly but better than duplicating files.


Feature idea: Public Schedules

For people who know how to host a CSV file, making a schedule available to others is pretty easy.  But many people don't now how to do that.  So here's an idea: What if the app could host the schedules?
I'm imagining something along these lines:

User creates a schedule and want to share it.
User clicks on button and gets a URL that can be used sent to others.
Behind the scenes, the schedule is stored somewhere (firebase?) so that it can be retrieved.
Ideally, it would be possible to resave to the same location (if desired) so that people can get updated schedules with the old URL.
URLs could simply be gibberish hashes, or users could have some control over the name of the schedule.

Some of this needs some for thought/fine tuning.  For example, we could always save the schedules and just provide the sharing link upon request.  That might require a user to identify a schedule (give it a name or something).  That could be done without a login system if we just required each new schedule to get a unique identifier, but it would make all schedules public (up to security via obscurity).  A login system would restrict access to ones own schedules (which might be listed, selected, deleted, etc.)
So there are lots of variations on this theme depending on what other sorts of things get added to the app.
If we end up doing anything else with permanent cloud storage, storing schedules should be relatively easy to add -- the big lift is setting up the storage mechanism (and perhaps user login) in the first place.  But I thought I'd drop this here so it doesn't get lost.

Google drive could work too.  Since this is likely to be addressed fairly late in the process, I'd go for the easiest solution unless @kvlinden has a preference in terms of future maintenance.

Google Drive would be my first choice because we don't need to worry about storage or security since it is handled by Google.

Google basically handles authentication in firebase as well.  But since our current schedule formats are csv, xlsx, or json, those might be easier to store in Drive.  Presumably people could also access the file outside of the app that way too.

I have no preference - FIrebase seems overkill for this CSV application.

Actually, I'd rather just be able to import my courses into my MS Calendar - that's where I put my public schedule.
And we could create a public lab calendar for the lab schedules.


Unable to sign In

After so many tries and different accounts, all i get is Unable to sign In in IOS 12.2. Anyone help me out on this?

Hey, did you figure this out? I’m having issues myself.

There's a bug right now. We're working on it. I will update this thread when it's fixed.

This has been fixed for a while now. Closing.


Handle too many requests when over api limit

If the worker gets a response from tinify that you're over the account limit, it should notifiy the user somehow (email probably) then put the current batch of images back into the queue.  right now we don't know if that the limit has been hit and the current batch of images gets stuck on "working".
This is not blocking as the user can always just click requeue for each image

Resolved


Rclone, better CI, and dont materialize nquads until all partitions are done

don't materialize the nq until all partitions are done
don't send to renci until the nq is generated (and thus all partitions are done)
implement rclone and lakefs clients for interacting with renci
fix a few tests in ci/cd
add tests for lakefs
add more docs
fix an env var for the geoconnex sitemap that had a hash inside of it which messed up the url

If you see a lot commit spam in the git log it is because I had issues getting the env vars passed through. It should be working now

just added a gleaner threads env var as well


fix(graph): graph page error

SUMMARY
When creating a Graph Chart with the feature flag ENABLE_EXPLORE_DRAG_AND_DROP  activated, the application freezes on a white screen.
Scope
The purpose of this PR is to resolve the bug with as little impact as possible on the application
Debuging
When a chart (Graph) is created several error messages are displayed on the console. One of the messages is
Warning: Failed prop type: Invalid prop `options` of type `object` supplied to `SelectControl`, expected `array`.

Proposed solution
As the interface is not very clear, and for some reason the contract was changed, the idea of the solution is to change this prop only in the case where the error occurred and not impact any other part of the application that may be using this data.
For this, only the src/explore/components/Control.tsx component was changed, transforming the options props into an array only for the use of SelectControl.
AFTER

TEST PLAN
No behavior has been changed, so the application should work normally. In addition, all tests already implemented should pass correctly.
To test if this fix is working correctly, it is necessary to activate the feature flag:ENABLE_EXPLORE_DRAG_AND_DROP
(The activation is not in this PR because it is not intended to change any application's behavior.)
ADDITIONAL INFORMATION
The bug was reported in this comment: https://github.com/apache/superset/pull/13210#issuecomment-788097589
@junlincc @ktmud

@kgabryje @zhaoyongjie in case you have time to review :)

@rusackas @yardz
I guess we need to fix controlPanel for GraphViz in the superset-ui repo. the columnSelectControl no adaptation DND in explore.
https://github.com/apache-superset/superset-ui/blob/54cee3aa95af344c83f73fb786966c778071a495/plugins/plugin-chart-echarts/src/Graph/controlPanel.tsx#L31-L47

I fixed the issue in the superset-ui repo
https://github.com/apache-superset/superset-ui/pull/991

I think we should try to avoid such specific prop manipulation logics in the general Control renderer. These type of errors should be fixed in each control itself instead.

I think we should try to avoid such specific prop manipulation logics in the general Control renderer. These type of errors should be fixed in each control itself instead.

As I don't understand the data flow of the application very well, I made the fix with the least possible impact but with the awareness that it is possibly not the correct way. As the fix was created in another PR, I will close this one because it is neither necessary nor correct.


Faster API with fragment caching?

I've previously tried to speed things up with eager loading of the includes. And it seems to work pretty well so far. And by patching the JSON renderer the view rendering has gone from like 20% of the profiling time to less than the threshold.
But now I've narrowed the biggest bottleneck down to "building the JSON". Basically 90% of the requests are spent in the recursive JSONAPI::ResourceSerializer#object_hash and related methods.
And this sometimes leaves a request log like this:
Completed 200 OK in 12242ms (Views: 278.9ms | ActiveRecord: 860.4ms)

Where it takes 860ms in the DB, 278ms rendering the JSON and 11104ms in "other", where my profiling points to the object_hash-methods mentioned above being almost all of the time spent.
Granted the queries get quite large in these rather complicated ?includes but this particular response wasn't even bigger than ~400KB so it still seems a bit too much with 12s. And most of the times these requests are caught in view caches and varnish so it doesn't really break our app yet. But I still would like to keep the uncached requests at most 1 second.
So considering this I was thinking maybe it could be quite "easy" to implement Rails fragment caching?
I haven't used it myself yet in a project (never had a reason too) but it seems to be real easy if you use something like jbuilder for generating your JSON.
In theory I'm naively thinking that it would be enough to just wrap these lines in a cache(source.cache_key)-block.
Anyone with some insight into what else could go wrong with this approach?

@slaskis Sorry for the long delay in looking at this. I've only briefly investigated this, so this is certainly not definitive.  While in a request (that has includes) there is a lot of time spent in the object_hash method, most of that time is spent in the ActiveRecord::Associations::CollectionAssociation#reader method. This is the rails code that gets and creates the (hopefully) eager loaded records.
We've recently fixed some bugs related to eager loading included resources, and not including ids if a resource is not included. Because of those two changes it's possible the code is now faster (by not loading has many related resources) then when you reported this issue. If you have some time would you mind looking at this again and providing an update?

Great!
Unfortunately I'm still quite a bit behind on that project, it's still on a fork of 0.3.3 which is before the finished spec so it requires updating a few consumers too. So it's hard to try that out in production at this moment.
But I have tried making a perf-branch for the peeps example app and it shows a slight decrease of object allocations, not a huge difference (~4%) but still an improvement. But that's just the simplest example of 10 contacts x 100 phone numbers.

Has there been any progress on caching so far? I tried it out myself and had some decent improvements in performance but there were a lot of tests failing cause of it

related issue #74

See merged PR #712


Use a shared enum for key prefixes

I need a way to query global state by prefixes for https://app.zenhub.com/workspaces/core-protocol-60953fafb1945f0011a3592d/issues/gh/casper-network/casper-node/4686. The existing code manually encodes byte arrays wherever a lookup by prefix was needed, but that's difficult to keep track of and it's pretty prone to errors, so I decided to create an enum that represents all possible key prefix combinations (excluding ones that are too broad, like querying all system/account/contract entities). This enum allows us to have a single data type for all the prefix lookups. Its deserializer is tested against the Key serializer to ensure that it remains binary compatible with the Key type.
This enum can also be a useful way to expose the capability of querying the GS by a prefix to binary port consumers in a safe way, because we can prevent them from making very large queries (like querying with just a key tag) and we can restrict which KeyPrefix variants we allow in the binary port.

bors r+

bors r+


Fuzzy Search scoring- sub-string scoring for less characters

This current method of scoring is just a little funky for users who dont understand the intention behind it.
if (allSubstringsContainedInCompareString)             {                 int count = query.Count(c => !char.IsWhiteSpace(c));                 int factor = count < 4 ? 10 : 5;                 score += factor * count;             }
if you search code for Visual Studio Code, you get nothing. But searching cod you get Visual Studio Code because more weight is given to number of characters matched is less than 4. @theClueless should this be removed?

Brief discussion of the above logic here https://github.com/jjw24/Wox/pull/112


[Experiment] Data validation with Soda

Related to #769.
TLDR: Easy-to-use framework. May not work with geospatial data checks.
Note, this PR has 2 commits: 1st commit is the one implementing Soda, and the write-up below focuses on the 1st commit; 2nd commit shows a new Soda feature using data contracts instead.
What
Implement data quality checks for bpl_libraries recipe in template-db dataset with Soda.
Checks:

title column: can't have duplicate and null values
region column: values should be within a defined set of values
wkb_geometry column: can't have null values and should have state projection (custom data check).

geospatial check didn't work

Where
Checks are performed in our Postgres db. Among many integrations with various sources, Soda can also be used on local files via dask or spark dataframe.
How
This section describes implementation from first commit.
Soda is very lightweight: it requires only 2 files for deploying data checks:

soda_sip/configuration.yml - contains db conn credentials: I used env variables to define them.

soda_sip/bpl_libraries_checks.yml - the actual data checks, aka expectations for bpl_libraries recipe. I also added more schema check examples that I found useful.

Running checks from CLI:
soda scan --data-source template_db --configuration ./configuration.yml bpl_libraries_checks.yml

An optional argument in the CLI command is saving output in a json file which can be viewed at soda_sip/output.json.
Other Soda notes

I like how easy and quick is defining data checks and the ability of creating custom data checks with a free-form SQL statement

I said above that Soda may not work with geospatial checks... In theory, it should because you can write free from SQL queries that are run in your engine of choice. However, I was getting a weird error:

I connected with Soda in their slack channel and they said it should work. They said I may have this issue.

Implementing Soda with local files via required dask dataframe is a bit annoying because it needs to be done in Python and dask uses presto SQL dialect under the hood

Regarding the data contract feature and how it's different from the main implementation:

allows to define a true data contract for a dataset with custom key-value pairs such as product owner, description, contact info, etc
all columns must be defined with an optional key whether a column is required.
as of now, must be invoked from Python

Update: geospatial checks work when you specify schema where geospatial functions live (typically in public):

All credit to @damonmcc 🎉


Adding DCO and CoC

Initial agreement of the group when deciding to move to the nodejs org was to not adopt a CoC, which at the time was within the rules as long as we'd discussed it.

Would like to hear from @kkoopa, @bnoordhuis and @agnat on this PR, perhaps they have no objections or can see a sensible path forward and it's just me that's causing trouble by even raising possible objections.

We have a DCO, it's in the README.  I don't particularly miss a CoC.

Ditto. Common sense should be enough.

I don't particularly miss a CoC either, but then members of the majority never do since they never feel excluded. So, I'm currently mildly in favor of merging this and be done with it.
At the very least we should add one line about what to do if you feel harassed/excluded/&c.

which at the time was within the rules as long as we'd discussed it.

Technically I don't think this was ever the policy, although you could argue that the policy has failed. Let me explain.
The process for WG's was documented to be "copy the contribution policy (which includes the CoC) to your WG." Of course we had no good language in there for individuals projects under WGs or projects that transfer into the project. But even in the WGs that were created almost none of them copied over those agreements and we've never effectively kept track. I wrote the policy and I don't think I've ever copied over these documents in the 40+ repos I've created here.
This is all to say that the process we designed hasn't worked and that copying or requiring writing a new policy in ever repo/WG is ineffective and we should find a better solution.
However, for legal reasons we do need the DCO to be in every repo and we actually need it to be in a CONTRIBUTING.md file because that's the file GitHub points users to when filing a pull request.
I'm starting to think a minimalist CoC that is foundation wide is a better approach to all of this. Several other foundations have done this and, because violations are rare, it's a nice service to have something to fall back on rather than having an up-front requirement we make every project and WG deal with.

My specific concerns are:

No politics
No agendas
No whining

Why did you close it? My point was that we are volunteering our spare time here for solving technical issues, I imagine none of us really wishes to deal with any extra drama.
If, say, a pull request were to be rejected based on merit, or lack thereof, I do not wish someone with hostile intentions to have fodder for suing me or hanging me out to dry based on hurt feelings. The current node.js code of conduct does not have safeguards for not abusing the code of conduct or harassing contributors with inane issues and imagined slights. This is what I meant with the above three bullet points. If they could be incorporated into the minimalist CoC you mentioned, I would probably not have further reservations.
The DCO is fine and can be put in CONTRIBUTING.md if it is more appropriate than having it in README.md.

I closed it because this PR won't be merged as is and I need to send a new one moving the DCO to the CONTRIBUTING file.

I imagine none of us really wishes to deal with any extra drama.

If you continue this line of commentary you will certainly attract drama, another reason I closed and locked it as this is the predictable outcome of this thread were it to continue.


Zweihander fixes

Changes / Comments
Changed the name of the translations.json to translation.json so that the en translations file can be created so that the translation keys are not showing in the released version of the character sheet.
Fixed 2 errors in the migrating attributes sheetworker:
Typo: "armor_Qualitiy" -> "armor_Quality"
Misaligned array comparison between:  "Movement_Current" and "movement_overage"
Roll20 Requests
Include the name of the sheet(s) you made changes to in the title.
Comments are very helpful for reviewing the code changes. Please answer the relevant questions below in your comment.

Is this a bug fix?
Does this add functional enhancements (new features or extending existing features) ?
Does this add or change functional aesthetics (such as layout or color scheme) ?
Are you intentionally changing more that one sheet? If so, which ones ?
If changing or removing attributes, what steps have you taken, if any, to preserve player data ?
If this is a new sheet, did you follow Building Character Sheets standards ?

If you do not know English. Please leave a comment in your native language.

Additional Fix, Roll Template {{keys}} were within the span for i18n translations. Due to there not being any translations the keys were not being triggered to show the results in the chat pane. added 4/13/2019.

I'll put a note to myself to double check this sheet after translations are ran tomorrow to ensure they are included this time.
Thanks,
Cassie

@nogileyelsew
Just fyi, your translations ran this time.


App Tour Feature in the App

Proposing an App Tour feature in the  app for improved user experience. Accessible via a "Take App Tour" button in settings, it provides a guided walkthrough covering satellite tracking, celestial events, and other features, offering quick info on each aspect of the app. Aimed at quick onboarding for new users and flexible exploration for experienced ones, the feature is designed for simplicity. Users can skip the tour if already familiar. The concise and engaging design allows for user feedback, fostering continuous improvement. Expected benefits include enhanced user satisfaction, increased engagement, and improved retention.

@andoriyaprashant assigned


How to limit number of clusters per class?

OrientDB Version: 2.2
How do I limit the number of clusters created per class?  My application, currently using Pyorient 1.5.5 to communicate with OrientDB 2.2, runs significantly slower on my 32 core production servers than it runs on my 4 core test VMs.  I suspect it is because OrientDB is splitting the data across more clusters when there are more cores.  Rather than let OrientDB set the number of clusters per class based on the number of processor cores, I would like to try setting a fixed number clusters per class to see if that improves performance for my app.  How do I limit the number of clusters per class?

Hi @thenesk
This is the command:
ALTER DATABASE MINIMUMCLUSTERS 4

(or just choose the number of clusters you prefer)
Thanks
Luigi

Thanks for the quick reply!  Would the same format command work for setting a maximum?
ALTER DATABASE MAXIMUMCLUSTERS 4

I think I have too many clusters on my production server, and would like to reduce the maximum number of clusters created per class

Hi @thenesk
just use MINIMUMCLUSTERS, it actually sets a maximum ;-)
Thanks
Luigi

That appears to have worked.  Feel free to close the issue.  Thanks!


Set unique class names plugin

Not all styles can be inlined, like those in At-Rules.
For even further scoping the left over CSS one could rename all selectors in a way that makes collisions with ones from other SVGs highly unlikely - by prefixing them with a manually passed string, by the basename of the SVG file or randomly generated.

Plugin PR ready: https://github.com/svg/svgo/pull/700


Tests for generated commands

This is a bit of a prototype currently, but it will want some discussions and this will be a good place for that.
The basic idea is to have a json file defining the commands in both json and serialized byte data. Then we either serialize the json, or deserialize the byte data depending on whether the command is one we send or receive.
This json file is being generated from my C# library (LibAtem) as that has tests checking it against the official SDK so should be correct.
Essentially, these tests verify that the command encoding matches LibAtem, but it doesn't directly guarantee they are correct, this is assuming that LibAtem ensures it has sufficient coverage against the SDK.
Some questions:

With the json structure of commands varying slightly across the libs, which side should the conforming code be on?
Should the json file only contain a whitelist of commands, that are implemented here?
The json file currently has 10 random testcases for each command, is that a good idea?
Perhaps it should be a file per command, otherwise the whole file will change if a single command needs regenerating

@baltedewit @Lange I expect you to both have some input on this

I'm not sure how to do the command serialization tests.
The rawName of the commands aren't exposed so it is possible to build a map of objects like is done for deserializing.
My thoughts are we could done one of

split each command into a serialize or deserialize. That would be breaking due to class name changes
expose the name used for serialization (I have tried this for AuxSourceCommand but it feels messy/confusing to expose a second name on each command
Something else?

With the json structure of commands varying slightly across the libs, which side should the conforming code be on?

I'd like do to as little work as possible inside the tests, so it makes sense to optimize the json file. It is not exactly LibAtem's responsibility though, so we can make a small standalone conformer script too.

Should the json file only contain a whitelist of commands, that are implemented here?

That doesn't matter too much. Ideally this library would contain all commands anyway. In the current situation it's just some extra lookups. On the other hand, not too hard to add if we go with the standalone conformer script.

The json file currently has 10 random testcases for each command, is that a good idea?

Sure, in case we have some mistakes regarding 8bit vs 16bit integers this might be nice.

Perhaps it should be a file per command, otherwise the whole file will change if a single command needs regenerating

Theoretically, yes. But would we regenerate that often and is that worth the extra bloat of the files?

The rawName of the commands aren't exposed so it is possible to build a map of objects like is done for deserializing.

Seperating out the commands wouldn't have to be a breaking change if you consider that the deserializing commands are technically part of the public API but they are only used internally. They are part of the public serializable commands purely for organizational purposes. Keeping current command names for serializable commands and new ones for deserialization would require no changes from library users.

With the json structure of commands varying slightly across the libs, which side should the conforming code be on?

It is not exactly LibAtem's responsibility though, so we can make a small standalone conformer script too.

It can go in the generator program then.
Although, I dont think all of it can be moved as the tests will need to know which id fields to compare as they sit outside of the properties object.

Should the json file only contain a whitelist of commands, that are implemented here?

That doesn't matter too much. Ideally this library would contain all commands anyway. In the current situation it's just some extra lookups. On the other hand, not too hard to add if we go with the standalone conformer script.

Having them all creates 'noise' in the test results, as it produces a testcase that doesnt do anything, which I am not that keen on. But not having them will make it a lot harder to add new commands with more chance of tests being forgotten/missed. Perhaps I can do a better job at filtering out the unsupported ones so they dont show up

Perhaps it should be a file per command, otherwise the whole file will change if a single command needs regenerating

Theoretically, yes. But would we regenerate that often and is that worth the extra bloat of the files?

Only when a bug is found in a command or something changed in one.

The rawName of the commands aren't exposed so it is possible to build a map of objects like is done for deserializing.

Seperating out the commands wouldn't have to be a breaking change if you consider that the deserializing commands are technically part of the public API but they are only used internally. They are part of the public serializable commands purely for organizational purposes. Keeping current command names for serializable commands and new ones for deserialization would require no changes from library users.

Can the user not subscribe to individual commands being received? If not then that makes this easy.

Another question is where should the generator code be hosted. Should it stay as its own repo?

Can the user not subscribe to individual commands being received? If not then that makes this easy.

The command is passed in the stateChanged event, so if a library user is using the class name in there that would be a breaking change, yeah
The other thing we can do is create a simple external mapping, but it would need to be maintained when commands are added.

ah, with the command to stateChanged then I dont think we can avoid this being breaking without making a bigger mess.
I'm not against a breaking change, but wanted to flag that before embarking on restructuring almost every command in case we wanted to avoid that

Can the user not subscribe to individual commands being received? If not then that makes this easy.

The command is passed in the stateChanged event, so if a library user is using the class name in there that would be a breaking change, yeah
The command is passed in the stateChanged event, so if a library user is using the class name in there that would be a breaking change, yeah
This is the case for Companion but its a select-case, so it would be quick to update that to new names.

@baltedewit I am finding there are a few different ways we are building the buffers in serialize methods. Any objection to standarising them to look like https://github.com/nrkno/tv-automation-atem-connection/blob/master/src/commands/Inputs/InputPropertiesCommand.ts#L57 ?
Im also thinking of moving the prefixing with the command name out of those methods, now that the name is exposed

Any objection to standarising them

Not at all, that is the way I build all new commands too. Just haven't gotten round to converting the rest.

I'm also thinking of moving the prefixing with the command name out of those methods

I don't see any problems with this at this point.

@baltedewit I think this is ready for a proper review now. There are some test 'failures' still, so I have disabled testing those commands for now and shall fix them up later on.
I havent moved the json mutation logic to the generator yet, as it cant all be moved. Some of it is to know what the id properties are, as they live outside of the properties object. Potentially this array could also be improved a bit if we allow sharing the converters between commands, but that can get very confusing if not done well. The ids might be simple enough to do with a single lookup table though (and if not perhaps they should be conformed to be more consistent)
The only remaining question is Where should the generator project be located?

Hi, sorry I haven't participated in this discussion yet. I've been very busy and making some life changes recently.
Things are beginning to settle down a bit, so let me know if my input is still requested. If so, a brief summary of the current status of things and what choices are left to be made would be very helpful!

Where should the generator project be located?

If we keep the mappings for atem-connection inside the test files then it might make more sense to keep it as a LibAtem library so anyone wanting to use it for another language can do so.
There's a also a few more commands that are still failing:

[ ] _pin: discrepancy between atem-connection and LibAtem in the enum
[ ] AMMO: Rounding error
[ ] KKFP: LibAtem incorrectly(?) uses doubles instead of integers for typings
[ ] CAMI: Rounding error
[ ] AMIP: Rounding error
[ ] FTSU: Unkown props getting overwritten by generator instead of ignored
[ ] TDdP: Range validation errors (250 should be max, but 251 is generated)

@Lange Not a problem, both @baltedewit and I have been off for a few weeks, so nothing has been blocked. I think the versioning PR is more likely to warrant some input if you want to join in, this one has some refactoring but its pretty straightforward so I am happy with its current state.

@baltedewit it might make more sense to keep it as a LibAtem library

Sure. Lets go with that for now, we can always move it later if we find it is becoming too atem-connection specific, or want to move the mappings out of the test files in here.
@baltedewit Be careful with fixing those failing tests, I dont remember exactly what the issues were, but I think I left them as they needed some testing to verify which side was at fault.


Fix config file in proj dirrectory

This resolves the issue raised in Issue #4142 where Molecule does not find the config.yml file the project directory {project_dir}/.config/molecule/config.yml but does find it in the home directory.
I ran tests by isolating the 2 functions being called when LOCAL_CONFIG variable is defined and verifying that the config file is found when in the project dir.  If there is no {project_dir}/.config/molecule/config.yml  then it looks in ~/.config/molecule/config.yml and finds the config file.
I have not been able to test the full project with the tox command as my environment is having issues.  The tox tests are not completing even on a freshly cloned repo.
The code in the find_vcs_root function could likely be modified more as it does not appear to be called elsewhere and it searches for ".git", ".hg", ".svn" dirs as well for what appears to be no reason as far as I can tell since it is only used for locating .config/molecule/config.yml
Let me know if anything can be improved upon here with my delivery of this as it is my first PR with the project.
Fixes: #4142

Closed due to being abandoned.


Fix embed suppression on non-self users when cache is enabled

This pull request provides a solution for the bug described in Issue #2644.

E ok, trimite te rog pull request-ul catre repository-ul original si pune un link aici.

https://github.com/serenity-rs/serenity/pull/2659


Implement API for working with the native browser assistant

Implement browser API function in the CoreApi

Add a specified rule
remove a specified rule
match rules against the specified url
is filtering enabled / change filtering state
is https filtering enabled / change https filtering state

Implement function for getting the original cert for the specified domain and ip
Proposed table schema:
Table1: certs
Table2: original_certs
Table3: domain_certs
domain_name, ip, cert_id, original_cert_id

Moved to milestone v1.5+extension-api


Fix AnimationPlayer method track call oneself with IMMEDIATE mode

Fixes #69681.
If AnimationPlayer method track calls oneself with Immediate mode, the process is interrupted and the state is not updated, so it is safer to update the state before the process.

Hi, I can confirm that this fixes one scenario.
However, there are others. There is another where the animation is played again if you try to exit via a method call towards the end, but in the second round, the call method tracks are ignored.

@LakshayaG73 Do you mean that you want to restart animation? That is not supported behavior currently, so I think it needs to be a proposal. See also this comment https://github.com/godotengine/godot/issues/64644#issuecomment-1336405961.

@LakshayaG73 The behavior is not clear, so PLEASE SEND MRP. Also it is going to be another issue.

So. I didn't need a new MRP.
Energy_MRP.zip
In this modified version of the same MRP, with your fix applied, both messages are not printed.
The only difference is that I have pushed the key in 'jump_post' to the end of the animation.

@LakshayaG73 I think it is a designed behavior, not a bug.
It seems that the animation player is stopped by the fact that the animation was played to the end after the play was called first in IMMEDIATE mode and then in the same thread.
It is dangerous to update oneself by IMMEDIATE mode in the first place. If there is no reason, you should use DEFERRED mode.

@LakshayaG73 If you are really concerned about the delay, use IMMEDIATE mode and extend the end of the animation slightly.
Alternatively, you can send a proposal to set the  method call mode "for each key". This way, you can use the DEFFERED mode only if you want to place the key at the end of the animation.

@LakshayaG73 Okay, I changed the location of the exit determination process. I think this solved your problem in Immediate mode, but we need to do a little testing to see if it affects other areas.

So there is one more issue.
Seems like if the next animation being played is a 'looping' animation, then it loops the current one once as well.
Uploading an MRP.

energy - Copy.zip

@LakshayaG73 Added a check that if an animation is changed in the middle of a process, it will abort the process that could rewrite the next animation. Please test it.

So after the most recent test, here are the findings:
1: This fixes the issue with non_looped animations looping.
2: This does not seem to fix the issue with function calls at the end of the animation, and the culprit seems to be this:

I cannot place the key where it needs to be. It only goes a bit forward, or behind. Never at the exact time needed.

Note that you might be able to force the value by typing it in the Inspector:

Select the key
Set its Time property in the AnimationTrackKeyEdit inspector

Note that you might be able to force the value by typing it in the Inspector:

I think it is not stable workaround at the moment (it seems that very small values, such as 0.0001, can be shifted...). I will look into it more later.

So, this latest version seems to have broken a basic scenario where the method is not being called at all, in the second animation, regardless of the time.
Can we go back a version?

@LakshayaG73 Which case is it? Can you send MRP again?

@LakshayaG73 Sorry, the queue may not have been functioning properly because the exit decision was corrupted. I believe this has now been fixed.

Still seeing the same thing with your latest commit.
The call function track gets completely defunct of the second animation.
The flow goes like this animation 1 -> transition via method call -> animation 2 -> transition via method call -> animation 3
For each of these, I was calling advance(0) after play(). Removing this advance(0) seems to get rid of the issue.

@LakshayaG73 That old commit was dangerous because the flag end_reached = true remained after the animation changed. advance(0) certainly removes that flag, but is probably hackey. If you remove advance(0), is there any more problem? If you still have problems, please send MRP.

Ok. I understand that we can't use that old commit. But there is some other issue here.
I'm saying that if I have the following:
func end_pre_jump():
	player.animationPlayer.play(animations[1])
	player.animationPlayer.advance(0)

This advance call does not go through, and essentially makes the played animation's tracks defunct.

@LakshayaG73 As far as using Immediate mode, that advance() is very dangerous.
Let me explain the flow.

The old animation's method track calls play().
play() rewrites the animation state to the new animation.
advance() processes and caches the new animation.
The old animation's method track is finished and the cache is deleted.

The problem is that the cache of the new animation is deleted at the time of 4. So advance() should not be called in such a case Immediate mode.
If deleting advance(0) solves the problem, I think this can be merged.

Ok, I think I understand...
Deleting advance() works well. Calling play() once more after advance() also solves the problem. Is that a viable fix?

The method track changing oneself in Immediate mode is quite dangerous, and essentially, it is not recommended to do so except in Differed mode. So I think there is nothing more that can be care on the core side.

Oh, the flow is a bit different: advanced() sets is_just_played() to false, so it seems that the old animation stop flag is revived. In any case, advanced() in Immediate mode is dangerous.

Oh, the flow is a bit different: advanced() sets is_just_played() to false, so it seems that the old animation stop flag is revived. In any case, advanced() in Immediate mode is dangerous.

Can this be fixed? : )

I'll take a look at it a little, but I think it's hard. If possible, I will send a follow up PR later. But just don't expect too much.

Thanks!

Seems that this change has broken my implementation now.
Case:
I have on timeline a function track with a function that is set to sec 0.8. In this function in code i do seek(0.2, true) on the timeline. So he repeats non stop the area on the timeline, until i say in code somewhere he should stop doing it, by jumping over the function (marker) that is for repating the part. With seek(0.85, true) for example
But it seems that this is not working any longer. He repeats and repeats regardless.

@TypeOverride The update option of seek() explicitly means to process that frame, so if there is a MethodTrack or AudioTrack on that frame, it will play it correctly. This is an intentional implementation, not a bug.
To avoid continuous firing, it is straightforward to implement seek on only one frame, or fire advance() on only one frame with update(0.2, false).

@TypeOverride Your problem is caused by your implementation approach and is not related to this PR. The update option of seek() means to process that frame in animation explicitly, so if there is a MethodTrack or AudioTrack on that frame, it will process those accurately. This is an intentional implementation, not a bug.
To avoid continuous firing, it is straightforward to fire seek(0.2, true) on only one frame, or fire advance(0) on only one frame after update(0.2, false).

no clue, but it not works anymore. So the Godot version before had a bug. But this "bug" was very natural then.
Calling a method on method track and jumping to certrain position. Not working anymore. For me it looks more like a bug now. Because i guess much people will try to do something like this, because its very logical to do.


Add cache_from option and bump to 1.1.2

This PR adds a new subsystem option cache_from that allows developers to specify a string representing the location of the cache registry.
Note that so far I've only tested this feature locally by configuring the cache_from option, confirming that the image continues to build, and that pants outputs the correct path to the configured cache manifest. I'm waiting on the Infrastructure team to provision a registry for us so that we can test this with a live remote registry.
$ ./pants package test_docker:dockerized_flask_app
...
#18 exporting cache
#18 preparing build cache for export done
#18 DONE 0.0s
------
 > importing cache manifest from us-docker.pkg.dev/sendwave-development/sendwave-docker/pants-docker/dockerized_flask_app:
------

@compyman I moved the cache_from parameter to a field on the target, pushed up a test image, and confirmed that this works 😎
$ ./pants package test_docker:dockerized_flask_app
...
#5 importing cache manifest from us-docker.pkg.dev/sendwave-development/sendwave-docker/pants-docker/dockerized_flask_app
#5 DONE 0.0s

#6 [internal] load build context
#6 transferring context: 122.70kB done
#6 DONE 0.0s

#7 [ 7/11] RUN python -m pip install --upgrade pip -c constraints.txt
#7 CACHED
...

Should be ready for a final look!


Accessibility : Highlighting errors in contact form

Description

[x] I have read the contributors guidelines .
[x] I have read the code of conduct.

In the contact us page, the toastify notification just mentions "Resolve Errors in the Form", but it doesn't highlight what are the errors in the form.  It would be helpful to the user if we can notify what are the errors in the form.
My approach to the problem:
* We can highlight the form entries which have incorrect type of information along with a single toastify notification.
Screenshots

Additional information
No response

It's great having you contribute to Milan ! This is an auto-generated textThe maintainers/owner will look into the issue soon, add relevant tags and proceed further, please have paitence. Meanwhile feel free to support star the repo and share it with your friends. :nerd_face: :rocket:

If you feel the issue to be worth pursuing, can you please assign me the issue ? Also if possible can you add the hacktoberfest tag?

Go for it !

Hi @IAmTamal thanks for assigning !! I have fixed the issue with three approaches:

Adding a toastify notification for each incorrect input.
Adding a  <p> element describing the issue in the form. with default toastify warn notification.
Adding both the above effects for the user.

Screenshots:

Using first approach:

Using second approach:

Using Third Approcah:

I like the second approcah beacuse it notifies the user what errors he made in the form along with one effective notification. Please tell me which approach should I implement while making PR.

Hello there 👋
I liked the way stuffs works out, however i would recommend something :
Case 1 :
If someone fills half/none of the form, then just display Fill out all the fields before sending

Case 2 :
If someone fills in all the field yet gives an invalid email, tell them to Please provide a correct email

Thanks for the speedy reply!! I have few doubts. So we don't need any instructions to be displayed on the page but only through toastify notification?  And also the notification should be condition based on the errors?

Thanks for the speedy reply!! I have few doubts. So we don't need any instructions to be displayed on the page but only through toastify notification? And also the notification should be condition based on the errors?

Hmm i think providing too much details is not of use, like if i have everything filled, but missed the last name  - it'll be still okay to say the user to Please fill out all the fields
So we have only 2 conditions  :

If any field is empty say Please fill out all the fields via toast !
If all fields are filled but email is invalid say Please provide a correct email

So I have implemented your suggestion. If any detail is missing except the email, then user gets Please fill out all the fields warn via toast and if the user fills all details correctly except the email then he gets Please provide a correct email warn via toast. So, I have not added any extra code in the page.

Screenshots:

User has not entered one or more detail:

User has entered everything correctly except the email:

Looks good, do make a PR once ready !

Cool! Will Make a PR soon!

Hi @IAmTamal what extra changes do I need to make in my PR?


[3.16.1] Switching IPIP to VXLAN error

On my on-premise cluster kubernetes (1.18), I deployed Calico 3.16.1 with manifest and configuration IPIP. As I want to use windows nodes, I have to switch to VXLAN configuration and it doesn't work.
I follow the documentation here : https://docs.projectcalico.org/getting-started/kubernetes/installation/config-options#switching-from-ip-in-ip-to-vxlan
As the pool has already been created, I used calicoctl to perform the change.
Expected Behavior
Calico nodes should restart with the good configuration and routes of each nodes should be ok
Current Behavior
Some nodes didn't take the right routing configuration
[cloud@ingress01 ~]$ netstat -rn
Table de routage IP du noyau
Destination     Passerelle      Genmask         Indic   MSS Fenêtre irtt Iface
<IP_ADDRESS>        x.y.72.193   <IP_ADDRESS>         UG        0 0          0 eth0
x.y.72.192   <IP_ADDRESS>         <IP_ADDRESS> U         0 0          0 eth0
<IP_ADDRESS> x.y.72.193   <IP_ADDRESS> UGH       0 0          0 eth0
<IP_ADDRESS>   <IP_ADDRESS>         <IP_ADDRESS> UH        0 0          0 cali8504ae9df2a
<IP_ADDRESS>      <IP_ADDRESS>         <IP_ADDRESS>     U         0 0          0 docker0

[cloud@ingress02 ~]$ netstat -rn
Table de routage IP du noyau
Destination     Passerelle      Genmask         Indic   MSS Fenêtre irtt Iface
<IP_ADDRESS>         x.y.72.225   <IP_ADDRESS>         UG        0 0          0 eth0
x.y.72.224   <IP_ADDRESS>         <IP_ADDRESS> U         0 0          0 eth0
<IP_ADDRESS> x.y.72.225   <IP_ADDRESS> UGH       0 0          0 eth0
<IP_ADDRESS>   <IP_ADDRESS>         <IP_ADDRESS> UH        0 0          0 cali6690c73972e
<IP_ADDRESS>   <IP_ADDRESS>   <IP_ADDRESS> UG        0 0          0 vxlan.calico
<IP_ADDRESS>   <IP_ADDRESS>   <IP_ADDRESS> UG        0 0          0 vxlan.calico
<IP_ADDRESS>   <IP_ADDRESS>   <IP_ADDRESS> UG        0 0          0 vxlan.calico
<IP_ADDRESS>    <IP_ADDRESS>    <IP_ADDRESS> UG        0 0          0 vxlan.calico
<IP_ADDRESS>  <IP_ADDRESS>  <IP_ADDRESS> UG        0 0          0 vxlan.calico
<IP_ADDRESS>  <IP_ADDRESS>  <IP_ADDRESS> UG        0 0          0 vxlan.calico
<IP_ADDRESS>    <IP_ADDRESS>    <IP_ADDRESS> UG        0 0          0 vxlan.calico
<IP_ADDRESS>  <IP_ADDRESS>  <IP_ADDRESS> UG        0 0          0 vxlan.calico
<IP_ADDRESS>      <IP_ADDRESS>         <IP_ADDRESS>     U         0 0          0 docker0

I also tried to get the calico-vxlan.yaml manifest and apply it. But the issue is still the same.
Possible Solution

Steps to Reproduce (for bugs)

Install calico 3.16.1 with IPIP configuration
Try to switch to VXLAN with calicoctl

Your Environment

Calico version : 3.16.1
Orchestrator version (e.g. kubernetes, mesos, rkt): On-premier 1.18.3
Operating System and version: CentOS Linux release 7.5.1804

@Maellooou could you share what your IP pools look like on this cluster?
e.g., calicoctl get ippools

@Maellooou I'm going to close this for now but if you're still running into this issue please reopen or file a new issue, thanks!


Update testing suite

Pull Request Checklist:

[ ] This PR addresses an already opened issue (for bug fixes / features)

This PR fixes issue #xyz

[x] Tests for the changes have been added (for bug fixes / features)
[x] Documentation has been added / updated (for bug fixes / features)
[x] HISTORY.rst has been updated (with summary of main changes)
[x] I have added my relevant user information to AUTHORS.md

What kind of change does this PR introduce?:

Removes some long-obsolete tests
Splits the conda CI job to test against xarray-stable and xarray-dev
Updates condaf CI jobs to use provision-with-micromamba with caching
Sets versions for pytest-loguru, sphinx_rtd_theme, and black that are more current
pytest is now always in colour
<EMAIL_ADDRESS>is now set until tox-conda supports<EMAIL_ADDRESS>

Does this PR introduce a breaking change?:

Not really, no.

Other information:

@Zeitsperre sounds good to use micromamba :) Does the test 3.9 fail due to incompatibilities with the latest xarray? Should the upstream test be allowed to fail?

@cehbrecht
That's precisely the reason why the conda-build is failing (I'm going to rename the tests to add the upstream flag for clarity here). I don't think the upstream build should be a blocker to merging a PR (it's something the maintainers need to keep an eye, not the contributors).
As it stands, the only thing blocking merge is an approving review. If we want to make some builds "required" in order to merge (like we have on xclim) that's something we could do.  Thoughts?

@Zeitsperre I thought we could have an xfail for the upstream test ...

@cehbrecht
Thanks for the approval. Merging now.
We can set things up that way (technically, it's already setup that way). If we set the build to continue-on-failure, We won't see the red X-mark any more (which defeats the purpose in my mind; we don't know if it failed unless we look for ourselves). Neither solution seems ideal.


Openload: RegexNotFoundError

Checklist

[x] I'm reporting a broken site support
[x] I've verified that I'm running youtube-dl version 2019.07.02
[x] I've checked that all provided URLs are alive and playable in a browser
[x] I've checked that all URLs and arguments with special characters are properly quoted or escaped
[x] I've searched the bugtracker for similar issues including closed ones

Verbose log

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'--verbose', u'https://openload.co/f/oYZN5SNgIhs']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2019.07.02
[debug] Python version 2.7.15+ (CPython) - Linux-4.15.0-54-generic-x86_64-with-Ubuntu-18.04-bionic
[debug] exe versions: ffmpeg 3.4.6, ffprobe 3.4.6, phantomjs 2.1.1
[debug] Proxy map: {}
[Openload] oYZN5SNgIhs: Downloading embed webpage
[Openload] oYZN5SNgIhs: Executing JS on webpage
ERROR: Unable to extract stream URL; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
Traceback (most recent call last):
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 796, in extract_info
    ie_result = ie.extract(url)
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 530, in extract
    ie_result = self._real_extract(url)
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/openload.py", line 429, in _real_extract
    'stream URL')
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 1005, in _search_regex
    raise RegexNotFoundError('Unable to extract %s' % _name)
RegexNotFoundError: Unable to extract stream URL; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.

Description

Probably nothing smart to say. Support for Openload got broken recently. Everything else is in verbose log.

> py -3.7 .\youtube_dl\__main__.py https://openload.co/f/oYZN5SNgIhs
[Openload] oYZN5SNgIhs: Downloading embed webpage
[Openload] oYZN5SNgIhs: Executing JS on webpage
[download] Destination: Shoujo Shuumatsu Ryokou - 12 [1080p]-oYZN5SNgIhs.mp4
[download]   0.2% of 248.95MiB at 1002.31KiB/s ETA 04:13
ERROR: Interrupted by user


Added black_box in layer_map benches (fix #3396)

Changes
Use criterion::black_box for more accurate benchmarking of the layer map data structure developed in #2998.
Fixes #3396
These changes regresses the performance measurements by 5-20%. However, this is quite small compared to the 70x performance increase introduced by #2998.

Merged, thanks @mattnappo!


"HTML5 form features" is a weird page, out of date

Hm. https://caniuse.com/forms
I'm not sure what to recommend to do about this page. It claims Firefox, Safari, Baidu and KaiOS have only partial support for "HTML5 forms", but there's no information about what is incomplete. There's nothing actionable about this.
I don't believe either Safari or Firefox have incomplete form implementations. But also have no idea when whatever was missing when this page was created shipped.
Should this page exist anymore? It feels old and abandoned. But also, it still counts towards the overall browser score on the homepage of caniuse.com. And it makes multiple browsers look ancient and out of touch.

Yeah, this one's a holdover of a long time ago when Opera announced a whole suite of new form controls. Because they were the first browser to release them all in one go they become a single feature on caniuse.
Since then I'd been updating it based on whether or not all of those sub-features were supported (see the Sub-features tab for the list). I think because the "month" and "week" input types aren't supported in Firefox & Safari the parent features are also only considered partially supported.
You're right though that such a page wouldn't be added today and I agree that it would be fine to remove it.

The feature no longer appears, so closing this ticket.


[WEB-1857] Use correct shipping location for checkout

📲 What
Use the location id from selectedShippingRule instead of project when calling createCheckoutInput.
🤔 Why
Some pledges in production had a shipping location set to a city or town (they should be set to a country) and that the incorrect shipping location matched the project location.
👀 See
The bug (user selected US Shipping, which is $30, no bonus support):

Jira
✅ Acceptance criteria

[ ] Late pledge a project on staging and select a shipping location that is different from the project location
[ ] Verify that the location of the created checkout and backing match the location selected in the previous step.

SwiftFormat found issues:

File
Rules

Library/ViewModels/ConfirmDetailsViewModel.swift:348:1
warning: (wrap) Wrap lines that exceed the specified maximum width.

Library/ViewModels/NoShippingConfirmDetailsViewModel.swift:289:1
warning: (wrap) Wrap lines that exceed the specified maximum width.

  Generated by :no_entry_sign: Danger


Colab pip install incompatibilities | Sparseml cant use yolov5 integration

Hello i'm a newbie to all of this so maybe im doing something wrong. Thanks in advance.
I've been using yolov5 training and detection for a while in a college project, and now i discovered neuralmagic integration with yolov5. I dont have access to a gpu so this sounded great. That said i use google colab, and after i clone the sparseml repository, the deepsparse one or even te neuralmagic/yolov5 fork into colab; and i try to pip install the requirements or launch the setup.py
I've watching all yolov5 related videos from your youtube channel, but i cant seem to replicate your results.
I tried a lot of things so i'm gonna add more than one issue in this post. Sorry if thats inconvenient.

Here is an example of the error i encounter, in colab:

!git clone https://github.com/neuralmagic/sparseml

!pip install sparseml[torchvision]

and here is the output:

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
arviz 0.15.1 requires setuptools>=60.0.0, but you have setuptools 59.5.0 which is incompatible.
cvxpy 1.3.1 requires setuptools>65.5.1, but you have setuptools 59.5.0 which is incompatible.
google-api-core 2.11.0 requires protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.19.5, but you have protobuf 3.20.1 which is incompatible.
google-cloud-bigquery 3.9.0 requires protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.19.5, but you have protobuf 3.20.1 which is incompatible.
google-cloud-bigquery-storage 2.19.1 requires protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.19.5, but you have protobuf 3.20.1 which is incompatible.
google-cloud-datastore 2.15.1 requires protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.19.5, but you have protobuf 3.20.1 which is incompatible.
google-cloud-firestore 2.11.0 requires protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.19.5, but you have protobuf 3.20.1 which is incompatible.
google-cloud-language 2.9.1 requires protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.19.5, but you have protobuf 3.20.1 which is incompatible.
google-cloud-translate 3.11.1 requires protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.19.5, but you have protobuf 3.20.1 which is incompatible.
googleapis-common-protos 1.59.0 requires protobuf!=3.20.0,!=3.20.1,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.19.5, but you have protobuf 3.20.1 which is incompatible.
tensorflow 2.12.0 requires numpy<1.24,>=1.22, but you have numpy 1.21.6 which is incompatible.
tensorflow 2.12.0 requires protobuf!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.20.3, but you have protobuf 3.20.1 which is incompatible.
tensorflow-metadata 1.13.1 requires protobuf<5,>=3.20.3, but you have protobuf 3.20.1 which is incompatible.
torchaudio 2.0.2+cu118 requires torch==2.0.1, but you have torch 1.12.0 which is incompatible.
torchdata 0.6.1 requires torch==2.0.1, but you have torch 1.12.0 which is incompatible.
torchtext 0.15.2 requires torch==2.0.1, but you have torch 1.12.0 which is incompatible.
Successfully installed GPUtil-1.4.0 click-8.0.4 gputils-1.0.6 jedi-0.18.2 jupyter-1.0.0 merge-args-0.1.5 numpy-1.21.6 onnx-1.12.0 protobuf-3.20.1 qtconsole-5.4.3 qtpy-2.3.1 setuptools-59.5.0 sparseml-1.4.4 sparsezoo-1.4.0 toposort-1.10 torch-1.12.0 torchvision-0.13.0
WARNING: The following packages were previously imported in this runtime:
[_distutils_hack,google,numpy,pkg_resources,setuptools]
You must restart the runtime in order to use newly installed versions.

In the videos i've seen , 10 months old, thats how you do it, but i cant get it to work. I think there is a version conflict in this modules [_distutils_hack,google,numpy,pkg_resources,setuptools]

In my laptop, no gpu, i've tried to use the train.py inside the neuralmagic/yolov5 repository like this:

python train.py --cfg models/yolov5s.yaml --recipe zoo:cv/detection/yolov5-s/pytorch/ultralytics/coco/pruned75_quant-none?recipe_type=transfer_learn --data Final_corregido.yaml --hyp ../yolov5-deepsparse-blogpost/yolov5-train/data/hyps/hyp.scratch.yaml --weights zoo:cv/detection/yolov5-s/pytorch/ultralytics/coco/pruned75_quant-none?recipe_type=transfer_learn --img 640 --batch-size 16 --epochs 5 --save-period 15 --optimizer SGD --project ../resultados --name yolov5s-sgd-pruned

And 10 hours later i thought it work, but now i have a .pt file that i cant load or export to onnx, with the basic ultralytics/yolov5 scripts,  so tried to install sparseml[torchvision] on my computer, no gpu, in a venv:

pip install sparseml[torchvision]

But when i try to use the comands, i get this error:

(env) C:\Users\shirk\IA\Sparseml>sparseml.yolov5.export_onnx --weights small_pruned.pt
C:\Users\shirk\IA\Sparseml\env\lib\site-packages\torchvision\models\detection\anchor_utils.py:63: UserWarning: Failed to initialize NumPy: module compiled against API version 0x10 but this version of numpy is 0xe (Triggered internally at  ..\torch\csrc\utils\tensor_numpy.cpp:68.)
device: torch.device = torch.device("cpu"),
Traceback (most recent call last):
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.3056.0_x64__qbz5n2kfra8p0\lib\runpy.py", line 196, in run_module_as_main
return run_code(code, main_globals, None,
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.3056.0_x64__qbz5n2kfra8p0\lib\runpy.py", line 86, in run_code
exec(code, run_globals)
File "C:\Users\shirk\IA\Sparseml\env\Scripts\sparseml.yolov5.export_onnx.exe_main.py", line 4, in
File "C:\Users\shirk\IA\Sparseml\env\lib\site-packages\sparseml\yolov5\scripts.py", line 16, in
from sparseml.pytorch.utils.distributed import record
File "C:\Users\shirk\IA\Sparseml\env\lib\site-packages\sparseml\pytorch_init.py", line 51, in
from .framework import detect_framework, framework_info, is_supported
File "C:\Users\shirk\IA\Sparseml\env\lib\site-packages\sparseml\pytorch\framework_init.py", line 24, in
from .info import *
File "C:\Users\shirk\IA\Sparseml\env\lib\site-packages\sparseml\pytorch\framework\info.py", line 26, in
from sparseml.pytorch.sparsification import sparsification_info
File "C:\Users\shirk\IA\Sparseml\env\lib\site-packages\sparseml\pytorch\sparsification_init_.py", line 24, in
from .distillation import *
File "C:\Users\shirk\IA\Sparseml\env\lib\site-packages\sparseml\pytorch\sparsification\distillation_init_.py", line 17, in
from .modifier_distillation import *
File "C:\Users\shirk\IA\Sparseml\env\lib\site-packages\sparseml\pytorch\sparsification\distillation\modifier_distillation.py", line 24, in
from sparseml.pytorch.sparsification.distillation.modifier_distillation_base import (
File "C:\Users\shirk\IA\Sparseml\env\lib\site-packages\sparseml\pytorch\sparsification\distillation\modifier_distillation_base.py", line 30, in
from sparseml.pytorch.sparsification.modifier import (
File "C:\Users\shirk\IA\Sparseml\env\lib\site-packages\sparseml\pytorch\sparsification\modifier.py", line 36, in
from sparseml.pytorch.utils import LOGGING_LEVELS, BaseLogger, LoggerManager
File "C:\Users\shirk\IA\Sparseml\env\lib\site-packages\sparseml\pytorch\utils_init_.py", line 24, in
from .exporter import *
File "C:\Users\shirk\IA\Sparseml\env\lib\site-packages\sparseml\pytorch\utils\exporter.py", line 37, in
from sparseml.onnx.utils import ONNXGraph
File "C:\Users\shirk\IA\Sparseml\env\lib\site-packages\sparseml\onnx_init_.py", line 22, in
from .benchmark import *
File "C:\Users\shirk\IA\Sparseml\env\lib\site-packages\sparseml\onnx\benchmark_init_.py", line 22, in
from .info import *
File "C:\Users\shirk\IA\Sparseml\env\lib\site-packages\sparseml\onnx\benchmark\info.py", line 28, in
from sparseml.onnx.framework import framework_info as get_framework_info
File "C:\Users\shirk\IA\Sparseml\env\lib\site-packages\sparseml\onnx\framework_init_.py", line 24, in
from .info import *
File "C:\Users\shirk\IA\Sparseml\env\lib\site-packages\sparseml\onnx\framework\info.py", line 26, in
from sparseml.onnx.sparsification import sparsification_info
File "C:\Users\shirk\IA\Sparseml\env\lib\site-packages\sparseml\onnx\sparsification_init_.py", line 24, in
from .analyzer import *
File "C:\Users\shirk\IA\Sparseml\env\lib\site-packages\sparseml\onnx\sparsification\analyzer.py", line 26, in
from sparseml.onnx.utils import DataLoader, DeepSparseAnalyzeModelRunner, ONNXGraph
File "C:\Users\shirk\IA\Sparseml\env\lib\site-packages\sparseml\onnx\utils_init_.py", line 25, in
from .loss import *
File "C:\Users\shirk\IA\Sparseml\env\lib\site-packages\sparseml\onnx\utils\loss.py", line 16, in
from scipy.stats import entropy
File "C:\Users\shirk\IA\Sparseml\env\lib\site-packages\scipy\stats_init_.py", line 485, in
from .stats_py import *
File "C:\Users\shirk\IA\Sparseml\env\lib\site-packages\scipy\stats_stats_py.py", line 39, in
from scipy.spatial.distance import cdist
File "C:\Users\shirk\IA\Sparseml\env\lib\site-packages\scipy\spatial_init.py", line 105, in
from ._kdtree import *
File "C:\Users\shirk\IA\Sparseml\env\lib\site-packages\scipy\spatial_kdtree.py", line 4, in
from ._ckdtree import cKDTree, cKDTreeNode
File "_ckdtree.pyx", line 1, in init scipy.spatial._ckdtree
ValueError: numpy.ndarray size changed, may indicate binary incompatibility. Expected 96 from C header, got 88 from PyObject

When i try to load the small_pruned.pt with:

model = torch.hub.load('ultralytics/yolov5', 'custom', path='./models/small_pruned.pt', force_reload=True)

i get this error:

Downloading: "https://github.com/ultralytics/yolov5/zipball/master" to C:\Users\shirk/.cache\torch\hub\master.zip
YOLOv5  2023-5-28 Python-3.10.11 torch-2.0.0+cpu CPU
Traceback (most recent call last):
File "C:\Users\shirk/.cache\torch\hub\ultralytics_yolov5_master\hubconf.py", line 49, in _create
model = DetectMultiBackend(path, device=device, fuse=autoshape)  # detection model
File "C:\Users\shirk/.cache\torch\hub\ultralytics_yolov5_master\models\common.py", line 344, in init
model = attempt_load(weights if isinstance(weights, list) else w, device=device, inplace=True, fuse=fuse)
File "C:\Users\shirk/.cache\torch\hub\ultralytics_yolov5_master\models\experimental.py", line 80, in attempt_load
ckpt = (ckpt.get('ema') or ckpt['model']).to(device).float()  # FP32 model
AttributeError: 'collections.OrderedDict' object has no attribute 'to'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\shirk/.cache\torch\hub\ultralytics_yolov5_master\hubconf.py", line 60, in _create
model = attempt_load(path, device=device, fuse=False)  # arbitrary model
File "C:\Users\shirk/.cache\torch\hub\ultralytics_yolov5_master\models\experimental.py", line 80, in attempt_load
ckpt = (ckpt.get('ema') or ckpt['model']).to(device).float()  # FP32 model
AttributeError: 'collections.OrderedDict' object has no attribute 'to'
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "C:\Users\shirk\OneDrive\Documentos\Master\Proyecto_II\GitHub\TP2\Coche\main.py", line 178, in
modelo_clasificador = torch.hub.load('ultralytics/yolov5', 'custom', path='./modelos/small_pruned.pt', force_reload=True)
File "C:\Users\shirk\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\torch\hub.py", line 558, in load
model = _load_local(repo_or_dir, model, *args, **kwargs)
File "C:\Users\shirk\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\torch\hub.py", line 587, in _load_local
model = entry(*args, **kwargs)
File "C:\Users\shirk/.cache\torch\hub\ultralytics_yolov5_master\hubconf.py", line 83, in custom
return _create(path, autoshape=autoshape, verbose=_verbose, device=device)
File "C:\Users\shirk/.cache\torch\hub\ultralytics_yolov5_master\hubconf.py", line 78, in _create
raise Exception(s) from e
Exception: 'collections.OrderedDict' object has no attribute 'to'. Cache may be out of date, try force_reload=True or see https://docs.ultralytics.com/yolov5/tutorials/pytorch_hub_model_loading for help

I found a solution to the colab problem. You can create virtual enviroments inside colab to avoid incompatibilities:

!pip3 install virtualenv
!virtualenv theanoEnv

the you:

!source /content/theanoEnv/bin/activate; pip3 install theano

And you can install sparseml without the previus issue:

!source /content/theanoEnv/bin/activate; pip3 install sparseml[torchvision]; pip3 list

now im using this command to prune my model:

!source /content/theanoEnv/bin/activate; sparseml.yolov5.train
--cfg yolov5/models_v5.0/yolov5s.yaml --recipe zoo:cv/detection/yolov5-s/pytorch/ultralytics/coco/pruned75_quant-none?recipe_type=transfer_learn
--data Final_corregido.yaml --hyp yolov5/data/hyps/hyp.finetune.yaml
--weights zoo:cv/detection/yolov5-s/pytorch/ultralytics/coco/pruned75_quant-none?recipe_type=transfer_learn
--img 640 --batch-size 64 --epochs 5 --save-period 15 --optimizer SGD
--project "/content/drive/MyDrive/IA/Pesos_yoloV5/Final_corregido" --name "Small_pruned"

This find is worth 2 full days of banging my head aginst the keybord.............

I will be back if im not able to use the .pt that comes out of this to export it or to use it directly. so dont close this please.

Now i'm trying to validate the model like this:

!source /content/theanoEnv/bin/activate; sparseml.yolov5.val
--task test
--data Final_corregido.yaml
--weights ../drive/MyDrive/IA/Pesos_yoloV5/Final_corregido/Small_pruned_quant_Blog/weights/best.pt
--conf-thres 0.40
--project ../drive/MyDrive/IA/Pesos_yoloV5/Final_corregido/Metricas --name Small_pruned_quant_Blog

And this is the output:

/bin/bash: sparseml.yolov5.val: command not found

But im able to use sparseml.yolov5.train and sparseml.yolov5.export_onnx without problems.

hi @MrShirker the correct command should be sparseml.yolov5.validation.  If you saw val somewhere in our documentation, we can get that updated.
What metrics are you getting back?

Hello @MrShirker
Just circling back to see if you are able to supply any metrics to continue the conversation? Thank you!
Jeannie / Neural Magic

Hi @MrShirker
Some time has passed between our last comment and hearing back. I'd like to go ahead and close out this thread. But if you would like to continue the conversation, please re-open this issue. Thank you so much!
Jeannie / Neural Magic


Remove base path on LineFormatter

OPT-IN, This makes smaller logs entries, it is easy to read too
Before
Error at /var/www/vhosts/myvhost.com/app/Models/MyModel.php:165)
[stacktrace]
 /var/www/vhosts/myvhost.com/vendor/laravel/framework/src/Illuminate/Database/Eloquent/Model.php(1475)
 /var/www/vhosts/myvhost.com/vendor/laravel/framework/src/Illuminate/Database/Eloquent/Builder.php(1218)
 /var/www/vhosts/myvhost.com/vendor/laravel/framework/src/Illuminate/Database/Eloquent/Builder.php(1199)

After
$formatter->basePath('/var/www/vhosts/myvhost.com'); //laravel has base_path()

Error at app/Models/MyModel.php:165)
[stacktrace]
 vendor/laravel/framework/src/Illuminate/Database/Eloquent/Model.php(1475)
 vendor/laravel/framework/src/Illuminate/Database/Eloquent/Builder.php(1218)
 vendor/laravel/framework/src/Illuminate/Database/Eloquent/Builder.php(1199)

@Seldaek ping

Thanks!


Combine radars

add code to combine multiple radar videos together.

Both are very good points. For the timestamp, I will just make it a flag and default it to off.
Regarding the timestamp mismatch, we need a better architecture for combining multiple data streams. I think we will need to open all stream at the same time, match across different streams, and then generate one single frame. Its a bit more involved and I will create a ticket for that.

Both are very good points. For the timestamp, I will just make it a flag and default it to off.
Regarding the timestamp mismatch, we need a better architecture for combining multiple data streams. I think we will need to open all stream at the same time, match across different streams, and then generate one single frame. Its a bit more involved and I will create a ticket for that.


Enable tsan on linux64 build

This pull request enables thread sanitiser for linux64 build (including unit tests & functional tests) and fixes/suppresses the following:

Switch ubsan to run on linux32 as tsan can't run there (linker does not support thread sanitizer)
Fix data race in CCoinJoinClientManager (triggered by concurrent access on fMixing)
Suppress data races triggered in zmq namespace (same suppression exists in bitcoin)
Suppress deadlock false positive the sme way bitcoin does it (i.e. deadlock:CChainState::ConnectTip). False positive description:

One thread:
generateBlocks|mining.cpp:124
    CreateNewBlock
        LOCK2(cs_main, cs_mempool) | called from miner.cpp:128
            TestBlockValidity | called from miner.cpp:235
                    ConnectBlock | called from validation.cpp:4200
                        ProcessSpecialTxsInBlock | called from validation.cpp:2123
                            […]
                                NotifyMasternodeListChanged
                                    UpdateCachesAndClean
                                        LOCK(cs_main, cs_governance) | called from governance/governance.cpp:352
Lock order: cs_main, cs_mempool, cs_governance

The other thread:
AddGovernanceObject
    LOCK2(cs_main, cs_governance) | called from governance/governance.cpp:293
        IsValidLocally | called from governance/governance.cpp:298
            […]
                GetTransaction
                    mempool.get(hash) | called from validation.cpp:893
                        LOCK(cs_mempool) | called from txmempool.cpp:1219
Lock order: cs_main, cs_governance, cs_mempool

This kind of lock order inversion looks like a bug in tsan (similar known issue: https://github.com/google/sanitizers/issues/814) and can be easily reproduced with 3 locks by changing the order of the last 2 locks: thread1 locks A, B, C and thread2 locks A, C, B (assuming a reversed unlocking order). Godbolt example here: https://godbolt.org/z/81Y6frYKv (same example with std::scoped_lock https://godbolt.org/z/175x6Yv9h)
I will keep the PR in draft until everything is green and passing in gitlab ci.

I question whether we should retain one run of functional tests that doesn't include any sanitizer so that it is nice and fast..

I question whether we should retain one run of functional tests that doesn't include any sanitizer so that it is nice and fast..

I could do that, but we need a target without sanitizers at all and a new job for testing as tsan and ubsan can't run together (see: https://github.com/google/sanitizers/issues/1106). We could use the linux64_cxx17 build target (with thread sanitizer removed from there) and create a new testing job derived from it, so we would have 3 testing jobs: linux32-test (ubsan), linux64-test (tsan), linux64-test-no-sanitizers (we can discuss to improve their namings).

I question whether we should retain one run of functional tests that doesn't include any sanitizer so that it is nice and fast..

I could do that, but we need a target without sanitizers at all and a new job for testing as tsan and ubsan can't run together (see: google/sanitizers#1106). We could use the linux64_cxx17 build target (with thread sanitizer removed from there) and create a new testing job derived from it, so we would have 3 testing jobs: linux32-test (ubsan), linux64-test (tsan), linux64-test-no-sanitizers (we can discuss to improve their namings).

Yes, I think it makes sense to have the following
linux64 - pure linux64, no sanitizers, unit and functional tests
linux64-thread - linux64, thread sanitizer, unit and functional tests
linux32-undef - linux32, undefined sanitizer, unit and functional tests

pls see 3748656bac (NOTE: I kept pure linux32 build/tests untouched just in case - they should not slow things down that much and we can always drop them later if needed)

feature_llmq_signing.py failed twice in ci so far..

I tried to reproduce locally, but after running the test that's failing for 2 hours in a loop I gave up. I increased the timeout of wait_for_sigs on the failing testcase (https://github.com/dashpay/dash/pull/4563/files#diff-78ac439373346c70f2c3236e1f34e274df611f2f2367bef84464bd0e59226940R106) and it does not fail anymore in the CI.


Support null values in a sorted map

For the CombineStrategy, we should support nulls in order to have double-delimiter when an input field value is null.
Looking for assistance on this stream routine to support null values:
    protected static Map<Integer, String> sortByKey(Map<Integer, String> map) {
        return map.entrySet().stream()
                .sorted(Map.Entry.comparingByKey())
                .collect(Collectors.toMap(Map.Entry::getKey, Map.Entry::getValue,
                        (oldValue, newValue) -> oldValue, LinkedHashMap::new));
    }

@apupier thoughts?

@mattrpav I have not read anything else but one question come to my mind when I read it. Why not using directly a SortedMap?
https://docs.oracle.com/javase/7/docs/api/java/util/SortedMap.html
To be sure, what can be null here are the value of the map provided as parameter?

I provided a PR to support null keys: https://github.com/atlasmap/atlasmap/pull/114
For null values here is a bug for Collectors.toMap : https://bugs.openjdk.java.net/browse/JDK-8148463
and i realize that in the title it is mentioning "null values" explicitely...
mightt need to get back to some "old" forEach way to handle the use case

I provided another PR supporting both null keys and null values: https://github.com/atlasmap/atlasmap/pull/115
it is using a TreeMap instead of Streams. there might a little performance overhead but seems the more readable code that we can have to handle null values.
What can be thought of is to use the TreeMap all the time to avoid the sort to happens several times. It really depends on the program logic to decide for that. I guess performance measure would need to be handle to be sure.

Yeah, SortedMap requires an impl, so I was hoping there was some streams magic that'd allow without it.

Java needs the "?" null check operator!

TreeMap sol'n merged.
Thanks!

Maybe Stream.ofNullable will help with Java 9 : https://bugs.openjdk.java.net/browse/JDK-8050819


B 18645 move grade to orders from allowances int 2

Second integration PR, integration was rebased from main before this got to main.
First integration PR
Main PR
Agility ticket
Summary
As part of previous work on the customer view, we moved the Grade and Current Duty Location fields to be associated with the Orders for all customer facing pages.
For the TOO/Services Counselor view, the pay grade is currently displayed under the Allowances section and edited by clicking the "Edit Allowances" button.
This work will be moving the pay grade to be under the Orders section rather than the Allowances section, as well as editing the pay grade by using the "View and edit orders" button.
Changes to be made for TOO/Services Counselor screens:

Update the pay grade to be shown under the Orders section.

Remove the pay grade from the Allowances section.

Update the "View and edit orders" button to allow editing of the pay grade.

Remove the editing of the pay grade from the "Edit Allowances" section.

How to test
TOO

Access the office app as a TOO
Make sure pay grade is under the Orders section
Make sure you can edit the pay grade
When you edit the pay grade, check that the Allowances -> Weight allowance was updated

Services counselor

Access the office app as a TOO
Make sure pay grade is under the Orders section
Make sure you can edit the pay grade
When you edit the pay grade, check that the Allowances -> Weight allowance was updated

Screenshots
Previous view:

Current view:

@traskowskycaci Quick manifest conflict

Should be fixed, at least until the next time it pops up!