proof of concept regarding merging rexport and pushshift data

Was able to get pushshift as mentioned in the README working to export old comments. Thought I'd mention it here.
It's possible to use pushshift to get data further back, but I'm not sure if it should be part of this project, since some of the older comments don't have the same JSON structure, I can only assume pushshift is getting data from multiple sources the further it goes back. It requires some normalization, like here and here.
The only data I was missing due to the 1000 limit on queries from using rexport were comments. It exported the last 1000 but I have about 5000 on reddit in total.
Regarding HPI:
Wrote a simple package to request/save that data, with a dal (whose PComment NamedTuple has similar @property attributes to rexports DAL), and a merge function, and now:
In [15]: from my.reddit import comments, _dal, pushshift_comments

In [16]: len(list(_dal().comments())) # from dal.py in rexport
Out[16]: 999

In [17]: len(list(pushshift_comments())) # from pushshift
Out[17]: 4891

In [18]: len(list(comments())) # merged data, using utc_time to remove duplicates
Out[18]: 4893

In [19]: comments
Out[19]: <function my.reddit.comments() -> Iterator[Union[rexport.dal.Comment, pushshift_comment_export.dal.PComment]]>

Its possible that one could write enough @property wrappers to handle the differences in the JSON representations of old pushshift data, unsure if thats something you want to pursue here.

total misclick there, sounds good.

Once I've merged changes from rexport and #83 into my fork...
Would be possible to merge some of this (pushishift). Would probably be optional, like:
@dataclass
class reddit(uconfig)
	...
    # path[s]/glob to one or more comment exports
    # https://github.com/seanbreckenridge/pushshift_comment_export
    pushshift_export_path: Optional[Paths]

....
# if user has set pushshift comment export path...
	from pushshift_comment_export.dal import PComment
	from pushshift_comment_export.dal import read_file

I can probably work on a PR at some point if thats something you're interested in.

Just merged #83 ! Let me know if you have any issues :)
Absolutely, would be great to merge!

Configs: IMO it's best to separate the configs, e.g. class reddit_pushshift, class reddit_rexport. Or even better?:
class reddit:
  class pushshift:
      export_path: Paths

  class rexport:
      export_path: Paths

That way:

the naming for export_path is consistent
if there are some extra config attributes, they wouldn't mix up, and less potential for clashes

Optional[Paths]: I thought that Optional[Paths] would look awkward (suggests it's sort of expected to be None). Considering that setting it to None would be fairly rare (you usually either use the module and set it to something reasonable, or don't have the config at all), I though that it would make sense to reserve empty string for that purpose: https://github.com/karlicoss/HPI/blob/1c20eb27aaf1c557c83c8582f13f905bf990b5a5/tests/get_files.py#L105

Code structure: I think ideally it's best to separate the different data providers into different files, extract the schemas/common stuff to something like common.py and have a minimal all.py file that merges them together, e.g.:

https://github.com/karlicoss/HPI/blob/master/my/github/all.py
https://github.com/karlicoss/HPI/blob/master/my/twitter/all.py

The idea is that if the user adds their own data source, they can benefit from common.py, and supply their own all.py (that's why it should be as minimal as possible). I think it will also make backwards compatibility easier. What do you think?

P.S. Just recalled, I also had some experiments/notes about possible ways of overriding/extending modules, will dig them out and post somewhere for the discussion.

any issues

all good on my end, been merging commits as you've been merging them into master; Though, I'm only using ghexport and rexport currently.

nested classes for config

Would that work with my.core.cfg.make_config? If so looks good. Would need to add some checks for backward compatibility. The avoiding clashes point is good as well, for the possibility of sharing secret files for data sources.

I though that it would make sense to reserve empty string for that purpose

Feels sort of wrong from a typing perspective, but simple enough. Though, if we're doing nested classes, just don't include the pushshift class if its not used.

all.py/common.py

Yeah, this is better organized, I'll follow how its done in github. I suppose following that pattern, in common.py, Instead of doing a try/catch to stop an ImportError for pushshift_comment_export.dal in the merge function, would be easiest to catch that in the data-source specific (my.reddit.pushshift.py) file, and return an empty list to the merge function if the user didn't provide config/export path.

Would that work with my.core.cfg.make_config?

I think so? As far as I understand, nested classes in Python are merely a syntactic thing (e.g. even if you want to access an attribute of the enclosing class, you'd have to specify a fully qualified name)

Would need to add some checks for backward compatibility

Yeah. Regarding backward compatibility, also thought that my/reddit/__init__.py would be handy then, so old imports of my.reddit continue working. However then my.reddit wouldn't be a namespace package anymore, and that means that overrides won't work. I guess if the individual modules were properly versioned, that would be easier to handle, could be a major version change, or something similar. I'll think more about it...

would be easiest to catch that in the data-source specific (my.reddit.pushshift.py) file, and return an empty list to the merge function if the user didn't provide config/export path.

Ah yeah. Ideally, if the config has empty path, the list of inputs is empty, and the module just emits nothing and everything just works. The issue now is that the modules (e.g. import ghexport) are usually imported on the top level, both because it's more convenient, and for type annotation purposes. That means that if the user never wanted to use it in the first place, it still crashes on import, so they either have to install the dependencies, or override all.py and comment out the modules they aren't using.
Something I can think on spot: reorganize all.py to something like:
from my.core import X
from .common import merge_events, Results

def gdpr():
    from . import gdpr as source
    return source

def ghexport():
    from . import ghexport as source
    return source

def events() -> Results:
    yield from merge_events(
        X(gdpr()    .events(), []),
        X(ghexport().events(), []),
    )

X is a function which adds some defensive behaviour, something like:
def X(factory, default):
    try:
        res = factory()
        return res
    except ImportError: # presumably means that the user hasn't installed the module. other exceptions would be legitimate errors?
        warn(f"Module {factory.__name__} is not configured blah blah blah")
        return default

, it could perhaps be somehow configurable, for example, respect disabled_modules and in that case it won't emit the warning?

a bit boilerplaty. Really wish python allowed lambdas with multiple statements... maybe alternatively could use importlib
this is not mypy transparent (although all.py is meant to be super short so maybe it's not a biggie)

I'll think about it more.. maybe I'll come up with a way of lazily importing module dependencies so it's mypy friendly.

X is a function which adds some defensive behavior, something like:

Recently created an all.py for my.location, leaving it here as reference
To keep this somewhat mypy-friendly, I ended up creating a top level 'Location' NamedTuple, even though each data source has its on 'Location' NamedTuple, though one could probably get around that by standardizing the interface/duck typing. Each source looks like:
def _google_locations() -> Iterator[Location]:
    yield from map(
        lambda gl: Location(lng=gl.lng, lat=gl.lat, dt=gl.at),
        filter(
            lambda g: isinstance(g, GoogleLocation),
            google_events(),
        ),
    )

... filtering, and then mapping it onto the common NamedTuple representation.
If someone was using this for the first time, I think the 'X' function which adds the fallback/warning message benefits outweigh the downsides of it being biolerplatey; If a user wanted to customize all.py to include their own sources, they could choose to exclude the check.
I can't think of any reason the 'X' function couldn't be abstracted away into core.

Really wish python allowed lambdas with multiple statements

yeah.... I've done some weird things before to try and get around this. elixir has spoiled me there.
https://github.com/seanbreckenridge/mint/blob/master/budget/budget/cleandata/transactions/__init__.py#L8-L16
https://github.com/seanbreckenridge/mint/blob/master/budget/budget/cleandata/transactions/transform.py#L28-L59
https://github.com/seanbreckenridge/mint/blob/master/budget/budget/cleandata/transactions/transform.py#L227-L242

though one could probably get around that by standardizing the interface/duck typing
Yep, or Protocol

If someone was using this for the first time, I think the 'X' function which adds the fallback/warning message benefits outweigh the downsides of it being biolerplatey

Good point! I guess having a helper in the core would also allow making it configurable, e.g. defensive + warning by default and error/throw in 'developer mode.

yeah.... I've done some weird things before to try and get around this
Huh, clever!

Just as an update, Will probably attempt to merge a PR using an all.py file to merge ghexport and pushshift comments w/ some core.warnings if either arent configured unless you've decided against another style here.

Yeah, I guess it makes sense -- people are more likely to start with just one data source, so would be nicer to make this easier. And then I or generally more paranoid people could have an explicit policy to always throw instead of handling defensively.
Also there is @warn_if_empty annotation which can be used to make it easier to catch if it's too defensive (guess later it can also respect some error handling policy to be stricter, e.g. throw).

similar problem described in the location issue I raised a while back -- where does the configuration for each module live?
I may be repeating points already said above or in other discussions, just wanted to get this down somewhere.
Ideally common.py shouldn't include imports to other relative modules -- it should just describe the common schema for some datasource, and have a function to merge data from each of them (and perhaps other shared code, but definitely shouldn't include provider (rexport, pushshift) specific configuration or code)
I'm leaning towards the following, as it means there is less problems overlaying/adding additional modules, and its slightly more mypy friendly -- as were not defining some expected nested configuration object (which the user may change if they add an additional source), with the two data sources here -- the only interaction all.py has with each module is trying to import the events/comments functions.
all.py
common.py
rexport.py
pushshift.py

common.py just includes the merge and shared NamedTuple model
rexport.py manages config like:
try:
    from my.config import reddit_rexport as user_config
except:
    from my.config import reddit as user_config
    # send deprecation warning

class rexport(user_config):
    export_path: Path

# handle migration/make_config and possible deprecations for the rexport configuration block

... so that any deprecation warnings/configuration issues specific to my.reddit.rexport are handled in that file, instead of all.py (since we'd want to keep all.py as minimal as possible, so there aren't issues with overlaying)
This does mean the configuration is a bit more verbose (reddit_rexport) as opposed to something which may inspect my.config.core.{enabled,disabled}_modules or do importlib magic to expose config/event functions, I think the tradeoff is worth it

agh. seems like pushshift is going through a bit of a restructure to improve their API/has been down a bit recently (i.e. now when I'm testing)
going to remind myself to go look back at here incase its still having issues in a while; may need to update pushshift_comment_export etc. etc. bit up in the air right now

Hmm. Yeah -- agree it would be nice to split up. The only thing to keep in mind is that backwards compatbility (with import my.reddit) would need to be implemented on call sites, because my.reddit can't have __init__.py? Unless you have some ideas how to work around it?
re: dataclasses -- right, yeah nestedness seems like it would be hard to handle.
Although I had a different structure in mind, which shouldn't lead to such issues?

user config
Kind of like I suggested here: https://github.com/karlicoss/HPI/issues/89#issuecomment-701667279
class reddit:
    class rexport:
        export_path = ...
    class pushshift:
        export_path = ...
    class gdpr:  # no GDPR atm, but just for the sake of demo
        export_path = ...

my.reddit.rexport: will have most of the code from current reddit.py? but instead of uconfig, inherit uconfig.reddit (with some compat code for older config format)

my.reddit.pushshift: inherit uconfig.pushshift

my.reddit.common

Ideally common.py shouldn't include imports to other relative modules -- it should just describe the common schema for some datasource, and have a function to merge data from each of them (and perhaps other shared code, but definitely shouldn't include provider (rexport, pushshift) specific configuration or code)

yep!

my.reddit.all
Kind of what I described here https://github.com/karlicoss/HPI/issues/89#issuecomment-701688805
from . import rexport
from . import pushshift
from . import gdpr

from .common import merge_events

comments = merge_events(rexport.comments, pushshift.comments, gdpr.comments)
upvotes  = merge_events(rexport.upvots  , pushshift.upvotes , gdpr.upvotes )
...

The exact details are debatable -- e.g. as we discussed about making it more defensive
So the user would override all.py if they want to mixin a new data source. A bit of copy pasting, but works?

my.reddit.rexport: will have most of the code from current reddit.py? but instead of uconfig, inherit uconfig.reddit (with some compat code for older config format)

ahh... for some reason I had it in my head that we needed to have the nested config as a dataclass somewhere and call make_config on it to handle migrations, but that isnt really necessary - that can just be handled in rexport.py. Even beter if we don't define some 'combined' configuration in the my.reddit module, as that leaves it more extendible for the user

my.reddit can't have init.py? Unless you have some ideas how to work around it?

yeah, don't think there is a workaround that allows overlaying code over my.reddit and something that is fully backwards compatible. Perhaps best way to handle this is to add an __init__.py file with a deprecation warning to update the import to my.reddit.all... and then actually deprecate it in a few months to convert it to a namespace package. If someone is very bothered by that they can always do the manual workaround of installing this repo as editable and removing the __init__.py file themselves in the mean time (though, if they'd know that was the error, I dunno)

override all.py to mixin a new data souce

yeah, that sounds good
I think all the patterns here work for the my.location issue I opened a few days ago as well

Cool! Guess would be nice to compile a proper doc eventually so we remember our own past discussions xD
Btw, even if either of class reddit.rexport and class reddit_rexport work now -- do you think there any clear potential pros and cons? The former is a bit nicer in terms of matching the hierarchy... although the latter is not that much more confusing. Not sure.

The nested config reminds me a bit of a nested JSON/YAML like object, which is typically used for configuration anyways, so it works well in that sense.
Also, another random though I had recently -- if the user is using the deprecated version of the config, we could probably use the repr and print something by doing:
for deprecated_attr in ["export_path", "export_dir"]...
...
high(f"""DEPRECATED! Please modify your config to look like:

class reddit:
    class rexport:
        export_path = {repr(attrs[deprecated_attr])}
""")

would probably be a good idea to update the docs some, yeah... we've had some long discussions in PRs before. could try doing some PRs, would give me practice in org mode
am currently writing up a readme that expands a bit on the problems with editable/namespace packages (and their usage in HPI as a plugin system, by extension) as part of a naive script to re-order my easy-install.pth, can probably link to that in SETUP.org when discussing namespace packages

Also worth mentioning that this seems to exist now:
https://www.reddit.com/settings/data-request
seems pretty complete, may be able to use that instead of pushshift, unsure on the amount of each, will try and parse the GDPR export and see.
Would probably be combined like the github module, into reddit.gdpr?

Will leave this open for now, since my.reddit.gdpr is yet to be written, but a good chunk of this has been added in #179


Feature req: More padding to options on theme editor

I like being able to generate the theme and the update are great.
Looking at the padding property it would be nice if there were two sliders for padding. Vertical and horizontal. This will give more customisation of the padding, maybe all four sides should be sliders?

It exists already nvm


[question] change report currency

Hi @mivano
I was wondering if there is a filter or input parameter to define the report currency since I will need the report in USD instead EUR

This is a bit tricky; there is no explicit filter option on the cost API to specify a different currency, but you can request the cost in USD. This was already done, but not shown (and there was an incorrect mapping).
So in version 0.17.0 you will get the cost, the currency (which can be EUR) and the CostUSD field in the JSON output formatter.
Will this suffice?

This looks really good. Do you think this can be displayed in the markdown format as well?

It could, but I am wondering if it does not become confusing if there are two values displayed (a local currency and the USD version). For the JSON and CSV export it does not really matter, but the console, text, and markdown formatters are already overloaded with data.
What if I introduce a flag that forces the values to be outputted as USD instead? If false, it uses the values in the currency we get from the API; if true, it returns the USD version instead. Would that be okay for your use case, or are you looking for both values at the same time?

Hi @mivano,
actually this is the perfect approach. A flag that we can use to decide if we want the local currency or in USD sounds perfect

Implemented in version 0.19. Use the --useUSD true parameter to force the formatters to use the USD currency.

Hi @mivano this is working perfectly


DMA init bug

There is a bug in line 130 of main.c in DMA init procedure.
The DMA channel2 and channel7 provide MemoryBaseAddr by value but it should be reference.
Therefore:
There is:
130: DMA_InitStructure.DMA_MemoryBaseAddr = (uint32_t)WS2812_IO_High;
Should be:
130: DMA_InitStructure.DMA_MemoryBaseAddr = **&**WS2812_IO_High;
And similarly for line 162 -> &WS2812_IO_Low instead (uint32_t)WS2812_IO_Low.
Even with this bug, some of 16 lines driving the WS2812 could work correctly in my application. Usually when running the code, pins 0..2 work well, therefore when I ran the demo, everything seemed work well. But of course the content of register &0x0000 and &0xFFFF can change unpredictably and is final code dependent. Therefore someone can observe different behavior.

I got same result.
PB14 and other pins got reversed polarity with errors.
Hope I did see this issue earlier.


Consider using ts-pattern for pattern matching

ts-pattern looks like it might be a good way to do instanceof checks when deciding on how a CrewMember should react to a DomainEvent.
I should research this library.

ts-pattern instanceof check has the exact same limitation as regular instanceof check and doesn't offer a viable alternative to current implementation.


Docs: fix link

Good catch, thanks! I'll rebase this and point this into the latest feature branch so that the lint tests are working, they're broken because of a library update not properly following semver, and there is already a fix for it in a PR.


Update Debugging.md

At least for me I needed to press F2 in Genymotion.

@facebook-github-bot shipit


Enhance resource inspector to understand LWRP files containing multiple resources

This enhances chef-resource-inspector to understand LWRP files which contain multiple resources.  This allows for using the resource inspector in documenation flows where these resources have been generated outside chef, as an example
Description
This previously, the resource inspector would just return one empty resource for a file that looks like this obviously synthetic example:
class Dummy1 < Chef::Resource::LWRPBase
  resource_name :dummy1
  description "A dummy resource"
  property :first, String, description: "My First Property", introduced: "14.0"
end

class Dummy2 < Chef::Resource::LWRPBase
  resource_name :dummy2
  description "Another dummy resource"
  property :second, String, description: "My Second Property", introduced: "14.0"
end

before it would return:
{
  "cook_foo": {
    "description": null,
    "default_action": [
      "nothing"
    ],
    "actions": {
      "nothing": null
    },
    "examples": null,
    "introduced": null,
    "preview": false,
    "properties": [
       (standard properties only)
   ]
}

With this change, it returns:
{
  "dummy2": {
    "description": "Another dummy resource",
    "default_action": [
      "nothing"
    ],
    "actions": {
      "nothing": null
    },
    "examples": null,
    "introduced": null,
    "preview": false,
    "properties": [
    (standard omittted)
         {
        "name": "second",
        "description": "My Second Property",
        "introduced": "14.0",
        "is": "String",
        "deprecated": false,
        "required": false,
        "default": null,
        "name_property": false,
        "equal_to": [

        ]
 ...
 and
   "dummy1": {
    "description": "A dummy resource",
    "default_action": [
      "nothing"
    ],
    "actions": {
      "nothing": null
    },
    "examples": null,
    "introduced": null,
    "preview": false,
    "properties": [
    (standard omitted again)
      {
        "name": "first",
        "description": "My First Property",
        "introduced": "14.0",
        "is": "String",
        "deprecated": false,
        "required": false,
        "default": null,
        "name_property": false,
        "equal_to": [

        ]
      }

Types of changes

[x] Bug fix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)
[ ] Chore (non-breaking change that does not add functionality or fix an issue)

Checklist:

[x] I have read the CONTRIBUTING document.
[x] I have run the pre-merge tests locally and they pass.
[ ] I have updated the documentation accordingly.
[x] I have added tests to cover my changes.
[ ] If Gemfile.lock has changed, I have used --conservative to do it and included the full output in the Description above.
[x] All new and existing tests passed.
[x] All commits have been signed-off for the Developer Certificate of Origin.

@jaymzjulian can you rebase against main?

@jaymzjulian can you rebase against main?

can do!  wish the test system luck...


Python Get angles like Android, missing Landmark info

Hi Google
I would like to calculate angles in Python but I noticed that Landmark Type info is missing in the API like it is possible in Android here
In the Python Pose.py I noticed that there is a Landmark class but it's not used.
Please help me get the angles. Thanks

You can get landmark x and y coordinates from MediaPipe Pose by the following:
 image_height, image_width, _ = image.shape
 # Convert the BGR image to RGB before processing.
 results = pose.process(cv2.cvtColor(image, cv2.COLOR_BGR2RGB))
nose_x = results.pose_landmarks.landmark[mp_holistic.PoseLandmark.NOSE].x * image_width
nose_y = results.pose_landmarks.landmark[mp_holistic.PoseLandmark.NOSE].y * image_height

Then,  you can get angle by following MLKit's example.


[AVM Module Issue]: DNS Zones - Deployment History limit

Check for previous/existing GitHub issues

[X] I have checked for previous/existing GitHub issues

Issue Type?
I'm not sure
Module Name
avm/res/network/dns-zone
(Optional) Module Version
No response
Description
I am deploying through the 'br/public:avm/res/network/dns-zone:0.2.4' module Dnszones
But running to the following problem:

I deploy seventy domains with records in it.
(Optional) Correlation Id
No response

Sorry about the delay on this one. Is @teemukom's comment accurate? If so, then that is likely the source of the issue no?
@aavdberg, can you take a look at the Deployment resources in the Resource Group (there is a "Deployments" tab in the table of contents of the Resource Group blade in the portal)? Does it eventually get pruned back down to 700? If so, then that suggests @AlexanderSehr is correct that we are not able to keep up with the pace of deployments getting created.
The fact that the deployment works with @batchSize(1) also suggests it's our inability to keep up.
@aavdberg if this is still occurring, can you share a recent correlation ID and we can investigate further?

I'm also curious what the var varPublicDnsZones looks like. That will help us understand how many sub-modules will be created for a given record.

Send you a ping in teams @alex-frankel

When using a batchsize(1) then it slows down the deployment and have the deployment history cleanup time to clean up the old deployments.

When using a batchsize(1) then it slows down the deployment and have the deployment history cleanup time to clean up the old deployments.

1 is maybe a bit ... low and 50 or 100 does the trick 😄

@alexandersehr, @aavdberg mentions that even with a batchSize of 10, he is running into this issue:

We did try it also with a batchsize of ten but it also fails because the automatic cleanup does not go fast enough.

Can we consider refactoring this module?

Just got off a call with @aavdberg and he is creating a large number of DNS records. With the current structure each new record (even if the record is a new module) will produce a unique deployment. All the deployments are ideally happening in parallel which is why the Deployment clean up process is not able to keep up.
@ChrisSidebotham / @AlexanderSehr / @jtracey93 -- can we consider refactoring this AVM module to not produce so many nested modules? While we ideally would be able to (and plan to) raise the limit, it won't happen in the short-term, so refactoring the module is the only short-term fix that I feel is practical.
The refactor would require creating all of the records in the entry point file for the module instead of in a nested deployment. I realize reduces the separation of concerns, but that might be a worthy tradeoff to increase usability for the AVM consumer. Thoughts?

Hola Mr @alex-frankel,
we could consider it - but - I want to provide some context that may / should influence the ultimate path. Bare with me 😄
As you may know, coming from CARML, we've traditionally always written modules in a way that they align closly with the Azure REST API's structure (i.e., how resource types are nested in provider namespaces and resource types may have child-resource types) and treated child-resource types as full-on modules that we published independently to their parent.
Now, since we migrated the library to AVM we don't do this anymore due to the technical limitations in the backend, that is certain manual updates that can make this quite painful for every new module. There is the hope to work through this eventually, but it's what we have today. We've already seen several issues being opened that asked for the child-modules, so there's definitely a need beyond what we're ourselves using in Microsoft & at customers.
Coming back to the issue at hand: The intuitive alignment with the API structure and the comprehensive publishing are the main drivers behind the design of the modules today - and doing so consistently. As a result, child-resource types are always nested in their own module file and are tested against the same rule set as their parent (with the sole exeption of the missing version.json file).
Some additional thoughts:
If we'd try to radically remove the amount of deployments in AVM modules we could do so essentially each time there is no nested loop. This would / should affect a lot of modules. But it would come at the price of making it impossible to publish them as their own modules.
The compromise would be to do so only for modules where you may not need a seperate published module. This may even be the case for the DNS Zone. Personally, I can't judge that even though I've seen a wide range of customer setups that slice and dice the permissions they grant in very specific ways. The other problem is that it would blow a big whole into the guidelines, compliance tests & consistency, effektively requiring to debate every resource type of every resource.
Personally, I value consistency a lot as its benefits usually massively outweigh the benefits of inconsistency. I can't however argue with the fact that in this case it comes at the very unfortuante cost of deployment speed. So much for my personal thoughts on the matter. I'm sure others have some thoughts too :)

Ultimately, your call :)
I don't think the deployment speed fix is a meaningful fix to this issue. That being said, @anthony-c-martin what is the status on the rollout for the performance improvements with nested modules?

Ultimately, your call :)
I don't think the deployment speed fix is a meaningful fix to this issue. That being said, @anthony-c-martin what is the status on the rollout for the performance improvements with nested modules?

That's true. The 'speed' is an issue by itself - while this issue is about the number of deployments and their cleanup. May have gotten a bit dragged away towards the end 😏  Will update

@aavdberg as an aside, if you'd like to be onboarded to the Deployments performance improvement that @AlexanderSehr referenced, we can onboard you manually. Just email me your subscription ID(s). We are in the process of rolling it out to everyone, but we are going to do that slowly and progressively.

@alex-frankel send you ping in Teams with subscription id

Hi @alex-frankel,
Is the deployment performance improvement open to anyone?


onSuccess is not executed if already cached

I want get the uiimage of remote and save it to the photo library, onSuccess is a  good solution, but I found that onSuccess is not executed if already cached, I cannot get the uiimage from here every time. Is this a issue or is there another solution for this?
WebImage(url: URL(string: "https://nokiatech.github.io/heif/content/images/ski_jump_1440x960.heic"))
        .onSuccess { image, cacheType in
            // Success
            print("\(image), \(cacheType)")
        }

and cacheType = SDImageCacheType?
20-02-01 16:11:19 CardView[37], ContentView: <UIImage:0x2830f3060 anonymous {1310, 256}>, SDImageCacheType

Seems because of hitting image cache, which is Sync. So by the time you access .onSuccess, the completionBlock already been called. 😓
I'll provide a quick fix, to allows async dispatch for manager. Maybe another beta version ?

Seems we should not make it async, because SwiftUI may like to create another new Struct when display changes, such as entering the Edit Mode of list.
I'll find the solution soon.

@dreampiggy thanks very much. I using SDWebImageSwiftUI by Swift Package, so I must wait for your next tag.

@foolbear Does this test case cover you use case ?
I test this fix the issue. Also, compatible with List Entering Edit Mode. (This is a edge case that SwiftUI create your View struct without calling onAppears...)

I using SDWebImageSwiftUI by Swift Package, so I must wait for your next tag.

This is not related to the package manager... For SwiftPM, you can choose to use Git Tag/Git Branch/Git Commit/ Local 4 types of dependency.
You can just clone this repo, checkout the branch, then drag the folder into your Xcode Project, Xcode will override the remote dependency with the local one, you can test and debug.
See: https://developer.apple.com/documentation/swift_packages/editing_a_package_dependency_as_a_local_package?language=objc

@foolbear If you previouslly use 0.x version (0.10.2 or somthing), I've already released one patch version v0.10.3 for this.
Note on the middle or end of February, I decide to release official 1.0.0 version, which break some of APIs from 0.x and provide a better support for WebImage on animation...
So, currently small patches will pick back to 0.x version, but after 1.0.0 released, no more changes will be applied on 0.x version.

@dreampiggy in v0.10.3, onSuccess executed every time. but warning occurs : "Modifying state during view update, this will cause undefined behavior.", and I cannot get the UIImage. Maybe I use it in a wrong way:
import SDWebImageSwiftUI

struct TestImageView: View {
    @State private var uiImageFromRemote: UIImage? = nil

    var body: some View {
        VStack {
            WebImage(url: URL(string: "https://avatars0.githubusercontent.com/u/827316?s=88&v=4"))
                .onSuccess { image, cacheType in
                    self.uiImageFromRemote = image
            }
            .resizable()
            .indicator(.activity)
            .transition(.fade)
            .scaledToFit()
            uiImageFromRemote.map { uiimage in
                Button(action: { /*to save uiImageFromRemote to photo library*/ }) {
                    Image(uiImage: uiimage).frame(width: 100, height: 100)
                }
            }
        }
    }
}

image for button not shown even adding DispatchQueue.main.async{}.
WebImage(url: URL(string: "https://avatars0.githubusercontent.com/u/827316?s=88&v=4"))
                .onSuccess { image, cacheType in
                    DispatchQueue.main.async {
                        self.uiImageFromRemote = image
                    }
            }

The sample code works well. Do you have something extra config (like WebP?) ? I can not reproduce your problem after that PR #77 was merged.

If you means "I want to update another View based on the onSuccess callback". Sadly, you business SwiftUI code is wrong.
Anything trigger the update value (even the value is same, pay attention), SwiftUI will call your body again.
To correct use do what you want, check this sample code:

@dreampiggy roger that and thanks!

I'll merge that and release a 1.0.0-beta2 version as well.


Fix python to C++ pointer conversion issue in moldrawFromQPainter

On 64-bit windows, I noticed that the value of ptr/pointer_to_QPainter can exceed the maximum unsigned long int value. In some cases, this caused an OverflowError when using MolDraw2DFromQPainter to make a PNG. I think that changing the ptr type to size_t should prevent this issue on all platforms, but there might be a better approach.
Here is the traceback I was getting:
    drawer = MolDraw2DFromQPainter(cpp_qp, options.width, options.height)
...\rdkit\Chem\Draw\__init__.py:177: in MolDraw2DFromQPainter
    d2d = rdMolDraw2DQt.MolDraw2DFromQPainter_(width, height, ptr, panelWidth, panelWidth)
E   OverflowError: Python int too large to convert to C unsigned long
--------------------- Captured stdout call -----------------------------
QPainter address: <PyQt5.QtGui.QPainter object at 0x000002A0856282E0>

Thanks @rachelnwalker!


direction on ARCOMMANDS_Generator_GenerateMiniDroneAnimationsFlip not work

when i specify any direction , minidrone did a front flip.
it's normal ?

Maybe the RollingSpider is in "wheel mode". You can send the command generated with ARCOMMANDS_Generator_GenerateMiniDroneSpeedSettingsWheels with present param set to 0.
In wheel mode, the RollingSpider will only make front and back flips.

Indeed ! thank you.


Joystick

With this you will be able to receive joystick buttons and axes events under Linux up to Lua.
Specific joystick Lua code is not part of this pull.

Check comments into joysticks_devjs.cpp to help you with driver installation and calibration

lua file: http://pastebin.com/R0Pfds9K


Warning about long formulas still relevant?

We still have a warning
Executing factory with a string of length  1948  > 1000, could trigger a bug: ...

in the code at
https://github.com/cms-analysis/HiggsAnalysis-CombinedLimit/blob/e7f355d2228a754c3b0a865912836e9e05f07f64/python/ModelTools.py#L95-L101
and around since 2012 https://github.com/cms-analysis/HiggsAnalysis-CombinedLimit/commit/25333c47cfef82f0e4d02b68cdd7e81d1565e0d1#diff-6aff0212eae895e959674ba1ef93b157076c49d94d89513acbb130b05f72d480R35-R36
Do we know if/when RooFit fixed this?

I don't know about this problem. That means it has either been resolved by now, or it was never reported to ROOT. I would bet it got resolved. Maybe try to just remove this check, and if there still is a problem then we fix it in ROOT?


Feat/Add description to delete season modal

The delete modal under seasons tab in the league dashboard needs to have a child component added.  This action cannot be undone.

@makkoli I can open a PR for this

Thanks, I went ahead and pulled.


Add CopyAppToSymbols to Compile-App

Useful functionality when repository has multiple apps.

Could you rename the parameter to CopyAppToSymbolsFolder - then I will accept the PR

Done

thanks


Stride setting in the last block with ResNet

I find that the stride in the last block in ResNet is set to 2 in this implementation, but is set to 1 in ruotianluo.
However, because the output size from RoI pooling is 7 as set in here, I think a stride of 2 may lead to a too small feature map after RCNN_top.
So which is better, a stride of 1 or 2?

thanks for pointing out this difference. I did not try stride=1. From what we got from stride=1 and ruotian's results, it seems that there is no difference, but I will compare two settings.

Hi @clavichord93 ,
I tried to set stride=1 on PASCAL_VOC, and it obtained a slight better performance than stride=2 when using roi crop-resize (0.756 vs. 0.754 at epoch=6). However, the computational time cost is slightly longer for stride=1 than stride=2. Hence, it depends on what we care more when setting the stride.


weighted operations: performance optimisations

There was a discussion on the performance of the weighted mean/ sum in terms of memory footprint but also speed, and there may indeed be some things that can be optimized. See the posts at the end of the PR. However, the optimal implementation will probably depend on the use case and some profiling will be required.
I'll just open an issue to keep track of this.
@seth-p

maybe relevant: #1995

%load_ext line_profiler

import numpy as np
import xarray as xr

from xarray.core.weighted import Weighted as w

shape_weights = (1000, 1000)
shape_data = (1000, 1000, 10)
add_nans = False

def lprun_weighted(shape_weights, shape_data, add_nans, skipna=None):

    weights = xr.DataArray(np.random.randn(*shape_weights))

    data = np.random.randn(*shape_data)

    # add approximately 25 % NaNs
    if add_nans:
        c = int(data.size * 0.25)
        data.ravel()[np.random.choice(data.size, c, replace=False)] = np.NaN

    data = xr.DataArray(data)

    return data.weighted(weights).mean(skipna=skipna)

%lprun -f w._reduce -f w._weighted_mean -f w._sum_of_weights -f w._weighted_sum -u 1e-03 lprun_weighted(shape_weights, shape_data, add_nans, skipna=None)

weighted(weights).mean(skipna=True) calls
https://github.com/pydata/xarray/blob/d1f7cb8fd95d588d3f7a7e90916c25747b90ad5a/xarray/core/weighted.py#L143
and
https://github.com/pydata/xarray/blob/d1f7cb8fd95d588d3f7a7e90916c25747b90ad5a/xarray/core/weighted.py#L130
da.fillna(0.0) in turn calls where(null(data), other). Thus notnull is called twice. This could be optimized by doing self.null = obj.null() in __init__(...). This might also allow self.any_null = self.null.any() to skip NaN handling if there are none. However, this needs more thinking if a Dataset is passed. Probably overkill but leaving this here for reference.


Provide an API to load the correct image resource variant on IJ

Images on IJ have a light (e.g., icon.svg) and a dark (e.g., icon_dark.svg) variant for each icon resource. We don't have a built-in way to load the right variant automatically, but it sounds like we should really have that built-in.

Already done


Add support for VoxelNeXt

Hi,
This is a pull request for VoxelNeXt. VoxelNeXt is a simple, and fully-sparse 3D object detector. The core idea is to predict objects directly upon sparse voxel features. This work has been accepted by CVPR 2023.
For this pull request, it contains
(1) backbone, head, detector in pcdet/models.
(2) argoverse2 dataset in pcdet/datasets.
(3) config files for nuScenes/Waymo/Argoverse2 datasets in tools/cfgs.
(4) updated README that includes the introduction of VoxelNeXt.
(5) some sparse loss functions in pcdet/utils/loss_utils.py.
(6) some utils in pcdet/models/model_utils/centernet_utils.py.
I have already tested both inference and training upon this code. Please feel free to contact me if there are any issues.
Regards,
Yukang Chen

Code for data preparation is here and respective documentation is here. Thanks for the amazing work!

@malicd Thank you for your reminder. We will also include these documentations in pcdet. Thanks Yukang @yukang2017 .

@malicd Thanks. We will merge them soon.

Note that it seems that there are some bugs for training VoxelNeXt on spconv 2.2/2.3 version. We are trying to fix it. Before we finish this, please use spconv in 2.1, e.g., spconv==2.1.25 for training!

Did this question solved?


cam should be initialized to zero matrix. Modify example images

Hope @jacobgil can merge this pull request. The modification seems legit to me.

Thanks, merged!


Add constant_char() rule to parser

It looks like the rule for con char was missed while making the parser.

For some reason the Haskell code skips char so we don't need and shouldn't do this. Can we close this? It's good that this gets documented though so thanks for making it.

Had a convo on Discord. We want to track this, but it looks like the official Haskell parser doesn't bother parsing con char either, so we don't feel like it's needed rn. Maybe when they add it.

seems like char is no longer in the spec


Allow locale with country in locale in users table

Right now, the users table has a column locale which unfortunately only accepts the locale and not the locale plus country. Working with internationalization means that we need to take into account not only the locale but also the country since there's a lot of information that depends on the country.
We should be able to enter en-ca, fr-ca, pt-pt, pt-br, etc

@alveshelio What is the use case for you? My initial thinking with locale was for languages only using ISO 639-1 codes but do you plan to also use it for location/region too?
Would you have different email templates for pt-pt and pt-br?


Question about how this spawns a server.

Hi, came across this project while looking into an idea of a project I have.
A simple matchmaking service for CS 1.6.
I am wondering where in your project are CSGO specifics, so I could attempt it modify it to meet HLDS requirements.
Does this project deploy a container when a match is organized?
anyway, impressive project, cheers!

after digging around more, it seems there is no orchestration in this, but rather has a definition of servers, it seems like no modification is needed in this regard, as it will try to match with whatever is listening on the defined hosts, on the specified ports, so it seems so far at least.

Hi, Yeah you are correct. Within this prototype you define your servers within the config file and are then given out to players.

I think some sort of container automation would be a really cool thing to implement, but im not exactly experienced with any of the tools you implemented, if I do figure out a way i'll be sure to send it back your way, thanks for getting back to me.


[BUG] Access violation using IBA::to_OpenCV() with an ImageCache-backed ImageBuf

Describe the bug
I get an Exception 0xc0000005 encountered at address ***: Access violation reading location 0x00000000 while trying to convert a HEIC image to OpenCV format. Here is the PoC I'm working with:
#include <algorithm>
#include <filesystem>
#include <format>
#include <iostream>
#include <ranges>
#include <string>
#include <vector>

#include <OpenImageIO/imagebuf.h>
#include <OpenImageIO/imagebufalgo.h>
#include <OpenImageIO/imageio.h>
#include <opencv2/core.hpp>
#include <opencv2/highgui.hpp>

#include "lib.hpp"

namespace fs = std::filesystem;
namespace rng = std::ranges;
namespace vws = std::ranges::views;

auto main() -> int
{
  // Get image paths
  auto images = std::vector<fs::path> {};
  std::transform(fs::directory_iterator("img/"),
                 fs::directory_iterator(),
                 std::back_inserter(images),
                 [](auto const& entry) { return entry.path(); });

  for (auto const& image : images) {
    auto loaded_image = OIIO::ImageBuf(image.string());
    if (not loaded_image.initialized()) {
      std::cout << loaded_image.geterror() << '\n';
      continue;
    }

    auto cv_image = cv::Mat {};
    try {
      OIIO::ImageBufAlgo::to_OpenCV(cv_image, loaded_image, {}, 1);
      cv::imshow(image.filename().string(), cv_image);
    }catch(const std::exception& e){
      std::cout << "Error when converting: " << e.what() << "\n";
      return -1;
    }
  }

  cv::waitKey();
  return 0;
}

To Reproduce

I'm using vcpkg on Windows with openimageio[libheif,opencv] as dependencies
In the folder I have .heic images taken with an iPhone

Expected behavior
I would expect to see the images shown via OpenCV highUI.
Evidence
When debugging I get a Exception 0xc0000005 encountered at address ***: Access violation reading location 0x00000000 exception.
Platform information:

OIIO branch/version: <IP_ADDRESS>
OS: Windows 11
C++ compiler: Microsoft (R) C/C++ Optimizing Compiler Version 19.35.32215 for x64
Any non-default build flags when you build OIIO:

Sorry this went so long without a response.
Do you run into this problem when reading any heic image? If only some of them trigger the exception, can you please supply one that can reproduce the problem?

Hi, no problem. Thanks for reaching out.
Yes, in general with heic images. I downloaded this test image (attached image1.zip) from here: https://github.com/tigranbs/test-heic-images.
I tried today with version <IP_ADDRESS>#2 (using vcpkg as toolchain) and the error persists.
Here is the minimal code to reproduce:
#include <iostream>
#include <string>
#include <OpenImageIO/imagebuf.h>
#include <OpenImageIO/imagebufalgo.h>
#include <opencv2/core.hpp>
#include <opencv2/highgui.hpp>

using namespace std::string_literals;

auto main() -> int
{
  auto file_path = "image1.heic"s;

  auto loaded_image = OIIO::ImageBuf(file_path);
  if (not loaded_image.initialized()){
    std::cout << loaded_image.geterror() <<  'n';
    return -1;
  }

  auto cv_image = cv::Mat{};
  try{
    OIIO::ImageBufAlgo::to_OpenCV(cv_image, loaded_image, {}, 1);
  }catch(const std::exception& e){
    std::cout << "Error when converting: " << e.what() << '\n';
    return -1;
  }

  cv::imshow(file_path, cv_image);
  cv::waitKey();

  return 0;
}

Thanks for the example. I'm unable to reproduce it on my end, it works fine. (In this case, I'm using OIIO master, on MacOS, with OpenCV 4.8.1).
The Access violation reading location 0x00000000 (that's a null pointer) has me suspicious that the cv::Mat never gets allocated because to_OpenCV is not doing anything at all.
I would like to suggest that you change the part of your program that calls to_OpenCV to include some error checking:
    bool ok = OIIO::ImageBufAlgo::to_OpenCV(cv_image, loaded_image, {}, 1);
    if (!ok){
        std::cout << "Error when converting: " << OIIO::geterror() << '\n';
        return -1;
    }

Does this print anything that might give us additional clues?

Ohh, that is strange 🤔
Funny thing is that the exception interrupts the program, it is not even caught by the try-catch statement.

I'll try again under WSL and see if I get different results using GCC. This could mean it is an environment issue with my Windows 11 and msvc configuration.
I'm attaching the full PoC project with vcpkg and cmake configuration. Probably I have something wrong with how I'm importing the packages.
test_heic.zip
I'll report back with my tests.

Hi, sorry for taking so long, I had issues with my dev environment.
I run it as you said and got the following stacktrace:
Thread 1 "test_heic" received signal SIGSEGV, Segmentation fault.
__memmove_avx_unaligned_erms () at ../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S:317
317     ../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S: No such file or directory.
(gdb) backtrace
#0  __memmove_avx_unaligned_erms () at ../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S:317
#1  0x00005555559e42cb in OpenImageIO_v2_4::copy_image (nchannels=3, width=3992, height=2992, depth=1, src=0x0,
    pixelsize=3, src_xstride=3, src_ystride=11976, src_zstride=0, dst=0x7ffff4d1b040, dst_xstride=3,
    dst_ystride=11976, dst_zstride=0)
    at /home/jorge/vcpkg/buildtrees/openimageio/src/v<IP_ADDRESS>-e861b9d6fc.clean/src/libOpenImageIO/imageio.cpp:842
#2  0x00005555559e396f in OpenImageIO_v2_4::convert_image (nchannels=3, width=3992, height=2992, depth=1, src=0x0,
    src_type=..., src_xstride=3, src_ystride=11976, src_zstride=0, dst=0x7ffff4d1b040, dst_type=..., dst_xstride=3,
    dst_ystride=11976, dst_zstride=0)
    at /home/jorge/vcpkg/buildtrees/openimageio/src/v<IP_ADDRESS>-e861b9d6fc.clean/src/libOpenImageIO/imageio.cpp:743
#3  0x00005555559e3f11 in OpenImageIO_v2_4::parallel_convert_image (nchannels=3, width=3992, height=2992, depth=1,
    src=0x0, src_type=..., src_xstride=3, src_ystride=11976, src_zstride=0, dst=0x7ffff4d1b040, dst_type=...,
    dst_xstride=3, dst_ystride=11976, dst_zstride=0, nthreads=1)
    at /home/jorge/vcpkg/buildtrees/openimageio/src/v<IP_ADDRESS>-e861b9d6fc.clean/src/libOpenImageIO/imageio.cpp:797
#4  0x00005555558f89fd in OpenImageIO_v2_4::parallel_convert_image (nchannels=3, width=3992, height=2992, depth=1,
    src=0x0, src_type=..., src_xstride=3, src_ystride=11976, src_zstride=0, dst=0x7ffff4d1b040, dst_type=...,
    dst_xstride=3, dst_ystride=11976, dst_zstride=0, nthreads=1)
    at /home/jorge/vcpkg/buildtrees/openimageio/src/v<IP_ADDRESS>-e861b9d6fc.clean/src/include/OpenImageIO/imageio.h:3060
#5  0x00005555558ef851 in OpenImageIO_v2_4::ImageBufAlgo::to_OpenCV (dst=..., src=..., roi=..., nthreads=1)
    at /home/jorge/vcpkg/buildtrees/openimageio/src/v<IP_ADDRESS>-e861b9d6fc.clean/src/libOpenImageIO/imagebufalgo_opencv.cpp:346
#6  0x000055555567d68f in main () at /home/jorge/test_heic/source/main.cpp:22

Ok, so I'm going through it step by step. When it reaches the copy_image function the src parameter is already NULL. Tracing back into the library I found this:
On imagebufalgo_opencv.cpp:
bool
ImageBufAlgo::to_OpenCV(cv::Mat& dst, const ImageBuf& src, ROI roi,
                        int nthreads)
{
   ...
   bool converted   = parallel_convert_image(
        chans, roi.width(), roi.height(), 1,
        src.pixeladdr(roi.xbegin, roi.ybegin, roi.zbegin, roi.chbegin),
        spec.format, spec.pixel_bytes(), spec.scanline_bytes(), 0, dst.ptr(),
        dstSpecFormat, pixelsize, linestep, 0, -1, -1, nthreads);
   ...
}

By this moment the source is still valid, but src.pixeladdr(...) is returning nullptr. Because of this part:
On imagebuf.cpp
void*
ImageBufImpl::pixeladdr(int x, int y, int z, int ch)
{
    validate_pixels();
    if (cachedpixels())
        return nullptr;
    x -= m_spec.x;
    y -= m_spec.y;
    z -= m_spec.z;
    size_t p = y * m_ystride + x * m_xstride + z * m_zstride
               + ch * m_channel_stride;
    return &(m_localpixels[p]);
}

cachedpixels() returns false, making the whole thing return nullptr and then the copy_image eventually fails.+
Nonetheless, adding loaded_image.read() before the conversion doesn't solve the issue.

The issue was solved adding this before the conversion step:
loaded_image.read(
      0,
      0,
      /*force=*/true
      );

For some reason the image is not being loaded on demand for HEIF in particular 🤔

Yes, in retrospect, this all makes sense! What's happening is that the IB is read lazily, and in this case it's backed underneath by an ImageCache. For such images, the pixeladdr() will be null, but of course, to_OpenCV needs the whole thing in the buffer. Duh. I will have a patch submitted shortly.

I think this is the correct fix: https://github.com/AcademySoftwareFoundation/OpenImageIO/pull/4013

Awesome! Thank you @lgritz 🙌

@jorgemb Are you able to test that patch on your end to verify that it fully solves it for you?

Hi! Yes, tested it locally and fully solved the issue for me.

Thanks, @jorgemb. This fix will be in the next OIIO release.


Available witness OOBI list from KERIA to smoothen UX

One of the user/UX challenges I see with wallets in particular is selection of witnesses and watchers. If every end user needs to know how to select witnesses then it will be a barrier to adoptability.
So I was considering the idea of KERIA announcing known witnesses it controls in its infrastructure and being able to update that list. Usage of non promiscuous witnesses for identifiers could be approved by the KERIA instance. This would lend itself to paid providers for KERIA with low user setup.
This does present some risks, particularly from a compromised KERIA updating the list of known witnesses. Maybe there are some mitigations here such as witnesses needing to sign this "updated list" etc, and having witnesses from other providers.
Curious to hear other people's thoughts.

One of the user/UX challenges I see with wallets in particular is selection of witnesses and watchers. If every end user needs to know how to select witnesses then it will be a barrier to adoptability.
So I was considering the idea of KERIA announcing known witnesses it controls in its infrastructure and being able to update that list. Usage of non promiscuous witnesses for identifiers could be approved by the KERIA instance. This would lend itself to paid providers for KERIA with low user setup.
This does present some risks, particularly from a compromised KERIA updating the list of known witnesses. Maybe there are some mitigations here such as witnesses needing to sign this "updated list" etc, and having witnesses from other providers.
Curious to hear other people's thoughts.

I like the idea. When you say 'known witnesses it controls' do you simply mean that the company that hosts the KERIA service, also hosts the witness services?
I think it is good that you called out witnesses from other providers since it is generally important but especially worthwhile for greater security, even more so in this case

Yeah, in the most basic case I meant that the company who hosts the KERIA also hosts the witnesses - this is probably not too different to what an enterprise experimenting with integrating KERI would do now in a pilot.
But yes, it would be a lot better to select witnesses from a mix of other providers to de-risk. Especially if witness rotation requires signatures from a threshold amount of prior witnesses.
Maybe this would be like a network of interconnected providers that use each other's witnesses (with some restrictions on load, or appropriate billing).

In KERI Witness rotation is not constrained by prior witnesses.  I.e only the controller signatures are needed to rotate witnesses not prior witness signatures. This enables a controller to recover from fully compromised witnesses.
The only requirement on a prior witness is that it publish the event that rotates it out. This provides a forwarding service to the new witness set.

Thanks @SmithSamuelM - this was brought up quite some time ago in a dev call (in reference to "Joint Confirmed Witness Rotation" in the whitepaper) and I thought it was going to be enforced. I may have misunderstood and conflated it with forwarding. But great, that makes things easier!
What security advantage is there then with 2FA witnesses, if an attacker who compromised the pre rotated keys can change all the witnesses?

Yes the "Joint Confirmed Witness Rotation" was a suggested optional feature. As the white paper indicates its comes with a significant drawback, as I meantioned above, of maintaining a majority of honest witnesses.
Given the added complexity and demands on controllers, it is not a required feature.
When adding a 2FA for a witness to published a fully signed event, the controller is protected even in the event that the controller's prerotated key(s) are compromised but not a threshold number of 2FA factors for the witnesses.  So this is an easier way to get a comparable level of security as joint witness confirmation.   An attacker has to compromise both a threshold number of controller prerotated keys and a threshold number of witnesses (2FA)  in order to publish a verifiable rotation event.

OK, thanks!
Regarding 2FA, perhaps I'm missing something. Without this feature, if all of the prerotated keys are compromised, can't the attacker just change the entire witness set to their own witnesses and publish this to watchers?
Or is this more related to it being more awkward for an attacker to spread the KELs to enough watchers? (if so would be interesting for watchers to be suspicious of complete swaps of witness sets and have a grace period for the controller to recover with the original witnesses)

Witnesses provide high availability so the controller doesn't have to interact with each and every downstream validator. They also provide detectability against compromised signing keys.
To clarify, without 2FA, witnesses do not provide any meaningful security against compromised pre-rotated keys. But with 2FA then the attacker has to compromise both the pre-rotated keys and the 2FA of a threshold number of witnesses.
Cooperatively delegated AIDs also provide protection against the compromise of pre-rotated keys of the delegate. Since the rotation event won't validate unless the Delegator approves it.
So the attacker has to compromise the Delegator in addition to compromising the Delegate.
And even if the attacker compromises the Delegator the first time, The delegator and delegate can recover from a compromised rotation with a recovery rotation. This recovery of a rotation with another rotation is not possible with non-delegated AIDs.
So if both the delegate and delegator use 2FA with their witness sets, then an attacker has to compromise,
Delegate's prerotated keys
Delegate's witness 2FA threshold
Delegator's signing keys
Delegator's witness 2FA threshold
And do it twice in a row to make it unrecoverable.

The main purposes of witnesses are

High availability of key state
Detectability of key state compromise

When witnesses are employed for non-delegated AIDs, any compromise of the signing keys of the controller MUST be published by the witnesses. This ensures the compromise is detectable, at the time of publication by the controller. Becasue no validator with accept an event that is not fully witnessed, the attacker can't exploit a compromise of the signing keys without being detected within the network latency of publication by the witnesses. No other security system, to my knowledge, has this assurance of near real time detectability.
The controller can then recover from the signing key compromise by issuing a recovery rotation that disputes the compromised signing keys and any events (ixn) that were so signed.
Compromise of pre-rotated keys does not have this protection. But pre-rotated keys are not needed to be exposed unless and until a rotation is required. This non-exposure means pre-rotated keys can be protected much better than signing keys. So the degree of difficulty of compromise of pre-rotated keys can be made arbitrarily high.
Therefore pre-rotated keys must be protected differently than signing keys. If pre-rotated keys are managed in such as way that they are just as easy to compromise as signing keys then this is an anti-pattern for using KERI.
Adding 2FA to the witnesses for all events provides an additional layer of security to both signing keys and pre-rotated keys. Or one could choose to only 2FA rotations and not 2FA interactions.

To clarify, without 2FA, witnesses do not provide any meaningful security against compromised pre-rotated keys. But with 2FA then the attacker has to compromise both the pre-rotated keys and the 2FA of a threshold number of witnesses.

This makes sense to me if the witness set doesn't change.
If the attacker:

compromises the pre-rotated keys of a non delegated identifier,
creates a rotation event that removes all of the original witnesses and inserts all new witnesses from the attacker

Doesn't this effectively skip the need to compromise the 2FA, since those witnesses are removed?

These are protections against live attack on the current key state. Witnesses also provide protection against dead attack on stale key state.  Once a witness has witnesses and event it has first seen, it can't unsee it. It will never witness a duplicitous event. So an attacker has to compromise the witness to get it to "see" or witness an alternative event history. This provides a layer of protection against dead attacks without 2FA.

Its easy to focus on live attacks and forget about dead attacks. This was the fatal flaw in did:TDW. They thought that all they had to care about were live attacks and "simplified" the method by dropping all the protections against dead attacks.

if the attacker compromises the pre-rotated keys of a non delegated identifier,
creates a rotation event that removes all of the original witnesses and inserts all new witnesses from the attacker
Doesn't this effectively skip the need to compromise the 2FA, since those witnesses are removed?

Or is this more related to it being more awkward for an attacker to spread the KELs to enough watchers? (if so would be interesting for watchers to be suspicious of complete swaps of witness sets and have a grace period for the controller to recover with the original witnesses)

@iFergal  Yes, you are correct. Sorry my answers were out of sync with these two questions and as a result, I need to caveat my list of protections to account for an attack that replaces all witnesses with fully compromised pre-rotated keys. I added a caveat in my list of protections above:
To clarify, as you point out, if the attacker compromises the pre-rotated keys and also rotates out all the witnesses and replaces them with witnesses of the attacker's choosing, then the attacker would have effectively bypassed the 2FA.
But this would be detectable by the controller unless the attacker also successfully eclipse attacks the watchers of the controller, and it should certainly be a cause for concern by any validator.  This means validators may want to wait before accepting such a rotation.
All watchers may also have to re-oobi to the new witnesses, which in itself could be problematic.
These sorts of attacks are where the Judge role becomes useful. Because the Judge can evaluate or appraise such rotations as problematic and insert a time delay as you suggest, for example.
Given that the controller detects that the compromised rotation has occurred, the controller can issue an alternate rotation i.e. duplicitous rotation, thereby creating duplicity that a Judge would then use as evidence against accepting the suspicious rotation. The Judge could then OOB reconcile the duplicity with the controller. Of course, this is of no help for validators that are not using watcher pools with Jurors and Judges. So best to protect one's pre-rotated keys very well to prevent this sort of compromise.
In summary, the 2FA on witnesses does not provide full protection, as in prevention, from compromise of the live pre-rotated keys for non-delegated AIDs but does force the attacker to replace a threshold majority of the witnesses which by itself is a detectable event that can be used by validators to not trust the rotation and used by the controller to issue a competing duplicitous event thereby creating duplicity that requires a successful eclipse attack to overcome.
Notwithstanding the former,  2FA does prevent live attack on the signing keys, because signing keys can only be used to authenticate IXN events which do not change witnesses. Since signing key compromise is the most likely compromise, then 2FA with the witnesses requries compromising both the signing keys and the 2FA in order to even publish a compromised ixn event. The controller then does not have to deal with the complication of publishing a recovery rotation that disputes compromised events which prevents any forks in the KEL.
As I indicated above, using delegation does provide a prevention not merely a detection recovery protection for compromised pre-rotated keys. And the 2FA on the Delegator's witnesses then comes into play.

Because KERI is about providing strong cryptographic protection, sometimes its easy to lose sight of the other considerations in an attack.
For example, replacing a full set of witnesses with witnesses controlled by the attacker in a single rotation, requires the attacker to stand up public infra-structure for those new witnesses. This comes at a cost. Because the witnesses have to be publically accessible by watchers, the attacker is traceable to that public infrastructure. This exposes the attacker to civil, criminal, etc liability and is in itself a disincentive to mount such an attack.
The vast majority of attacks take time for the attacker exploit, so they must remain undetected during this time, and un-traceable back to the attacker after detection.
But best to not have to depend on such protections.  So as I indicated above, if one does not differentially protect ones pre-rotated keys, then that is an anti-pattern for using KERI.

Thanks for the clarification @SmithSamuelM - makes complete sense!
In light of the original issue, I'd be interested to see if there would be a way to have a single 2FA with KERIA (if those particular witnesses have some trust in the KERIA instance), not sure. Not as secure as individually 2FA'ing with each witness but more adoptable from a UX perspective, which could be OK for less susceptible/critical identifiers.

Because the witnesses have to be publically accessible by watchers, the attacker is traceable to that public infrastructure. This exposes the attacker to civil, criminal, etc liability and is in itself a disincentive to mount such an attack.

Didn't think of this, very interesting!

Because every action in KERI must be signed with non-repudiable digital signatures, every action, including those taken by attackers, becomes provably traceable to the signer. Thus, attackers can't attack anonymously in Keri.
A single 2FA between a Keria instance and witnesses is a little more secure than no 2FA. I guess it depends on how one evaluates the cost/benefit tradeoff.
If it ends up being a shared secret then it probably would not deliver enough security and may give a false sense of more security than it actually does.
This is the bane of current security  systems. They bolt on protection measures that are at best little more than security theater and at worst may induce users to engage in riskier behavior.

Agreed, definitely want to avoid shared secrets.
I kind of meant more like a 2FA between just the controller and KERIA itself. And thereafter some signed propagation message from the KERIA agent containing the KEL (not a receipt) was received by witnesses in that infrastructure, they would receipt and publish.
So an attacker has to compromise (pre-rotated keys, KERIA) or (pre-rotated keys, 2FA) to keep the same witnesses. Not as good, but in some cases as you say could be OK given the cost/benefit analysis. Will think about it more though!

Closing this for now as #318 suffices for me, for now.


Clustering Benchmark

To reproduce the scVI paper, it is important to add (when labels are available) some clustering metrics to the benchmark.py file .

When cell-types are available, add the NMI, ARI and ASW (see scVI paper)
When continuous information is available (protein in CBMC or PBA in HEMATO), add the metrics from the scVI paper (overlap in kNN tree)

[[ I merged a bit fast but the build is passing on master (travis push build passes, but no travis pr build because of "git fetch origin" failure ]]


Possible bugs in greenkhorn algorithm

Describe the bug

Whenever greenkhorn is called with log=True, the algorithm will return error.
greenkhorn does not allow for list input (I have to convert a, b, M as np.array manually), while sinkhorn_knopp does (in sinkhorn_knopp, the first three lines of code convert list to np.asarray).

To Reproduce
Steps to reproduce the behavior:

a = [.5, .5]
b = [.5, .5]
M = [[0., 1.], [1., 0.]]
a, b, M = np.array(a), np.array(b), np.array(M)
(greenkhorn will not allow list object, and have to convert manually)
T, log = ot.bregman.greenkhorn(a, b, M, 0.001, log=True)
(this line encounters error)

Expected behavior
It seems that I can never return the log dictionary using greenkhorn, and I am not sure why.
When running original sinkhorn, the log argument works fine.
Besides, I hope greenkhorn could allow for list as input (this is not an bug, but maybe it could be added in the future).
Screenshots
If applicable, add screenshots to help explain your problem.
Desktop (please complete the following information):

OS: MacOSX
Python version [2.7,3.6]: 3.6
How was POT installed [source, pip, conda]: conda

Output of the following code snippet:
import platform; print(platform.platform())
import sys; print("Python", sys.version)
import numpy; print("NumPy", numpy.__version__)
import scipy; print("SciPy", scipy.__version__)
import ot; print("POT", ot.__version__)

import platform; print(platform.platform())
Darwin-18.2.0-x86_64-i386-64bit
import sys; print("Python", sys.version)
Python 3.6.7 |Anaconda custom (64-bit)| (default, Oct 23 2018, 14:01:38)
[GCC 4.2.1 Compatible Clang 4.0.1 (tags/RELEASE_401/final)]
import numpy; print("NumPy", numpy.version)
NumPy 1.15.4
import scipy; print("SciPy", scipy.version)
SciPy 1.1.0
import ot; print("POT", ot.version)
POT 0.5.1

Additional context
Add any other context about the problem here.

Hello,
Thank you for reporting this bug so precisely, I have a PR with a correction in the works here: #76. It works on the new test and should also work on your example. It will be merged in the following days but it might take more time to get into a proper release. Feel free to install directly with pip from the git of the PR branch if you want it now.

Thank you so much! The script in the PR works perfectly for me!


unable to get pubsub messages, 'pubsub:event' is not fired

I am working with pubsub on ionic framework, i'm able to publish message and call getItems from node. but 'pubsub:event' is not fired. i'm unable to get real-time message which is published on node by other users. my code is as below...
publish
this.client.publish("pubsub.dev1.internal", 'mynode', {
        json: {
          value: 'Any JSON content could go here'
        }
      }, (err, data) => {
        if (err) {
          console.log(err);
        } else {
          console.log(data)
        }
      });
  }

getItems
this.client.getItems("pubsub.dev1.internal", 'mynode', opts ,
(err, data) => {
        if (err) {
          console.log(err);
        } else {
          console.log(data)
        }
      })

pubsub:event
this.client.on('pubsub:event', function (msg) {
      console.log("pubsub:event");
      console.log(msg);
    });

messages which are published are stored in server and when i call getItems i can fetch that messages, but the problem is messages are not received  on real-time using 'pubsub:event'. can anyone help me to solve this?

Just to be sure, you did subscribe to that node first, right?
this.client.subscribeToNode("pubsub.dev1.internal", "mynode");

i have tried subscribeToNode as below
this.client.subscribeToNode("pubsub.dev1.internal", "mynode", function (err) {
      if (!err) {
        console.log('subscribed');
      }else{
        console.log('subscribe error');
        console.log(err);
      }
    });

but i on callback i got error like..
error:{condition: "timeout"}

added one second timeout before subscribing and its working..
setTimeout(() => {         this.subscribeTONode();     }, 1000);


9.12.0: Adding a snapshot file to a directory causes  TypeError: Key "rules": Key "import/no-extraneous-dependencies": Could not find plugin "import".

As part of normal dev work I added a snapshot (Jest) file to a workspace in my repo. This apparently broke Eslint. When I remove the snapshot file the TypeError goes away. See NOTE THIS LINE in the middle.
✗ yarn workspace foo lint:fix
yarn workspace v1.22.19
yarn run v1.22.19
$ eslint src --fix
(node:41899) ExperimentalWarning: Importing JSON modules is an experimental feature and might change at any time
(Use `node --trace-warnings ...` to show where the warning was created)

Oops! Something went wrong! :(

ESLint: 9.12.0

TypeError: Key "rules": Key "import/no-extraneous-dependencies": Could not find plugin "import".
    at throwRuleNotFoundError (/Users/justin.collum/Documents/work/bar/node_modules/eslint/lib/config/rule-validator.js:66:11)
    at RuleValidator.validate (/Users/justin.collum/Documents/work/bar/node_modules/eslint/lib/config/rule-validator.js:147:17)
    at new Config (/Users/justin.collum/Documents/work/bar/node_modules/eslint/lib/config/config.js:240:27)
    at [finalizeConfig] (/Users/justin.collum/Documents/work/bar/node_modules/eslint/lib/config/flat-config-array.js:216:16)
    at FlatConfigArray.getConfigWithStatus (/Users/justin.collum/Documents/work/bar/node_modules/@eslint/config-array/dist/cjs/index.cjs:1102:55)
    at FlatConfigArray.getConfig (/Users/justin.collum/Documents/work/bar/node_modules/@eslint/config-array/dist/cjs/index.cjs:1120:15)
    at entryFilter (/Users/justin.collum/Documents/work/bar/node_modules/eslint/lib/eslint/eslint-helpers.js:286:40)
    at async NodeHfs.<anonymous> (file:///Users/justin.collum/Documents/work/bar/node_modules/@humanfs/core/src/hfs.js:560:24)
    at async NodeHfs.<anonymous> (file:///Users/justin.collum/Documents/work/bar/node_modules/@humanfs/core/src/hfs.js:590:6)
    at async NodeHfs.walk (file:///Users/justin.collum/Documents/work/bar/node_modules/@humanfs/core/src/hfs.js:600:3)
error Command failed with exit code 2.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
error Command failed.
Exit code: 2
Command: /Users/justin.collum/.nvm/versions/node/v20.14.0/bin/node
Arguments: /opt/homebrew/Cellar/yarn/1.22.19/libexec/lib/cli.js lint:fix
Directory: /Users/justin.collum/Documents/work/bar/lambdas/cawo/queries-resolver
Output:

info Visit https://yarnpkg.com/en/docs/cli/workspace for documentation about this command.

✗ rm lambdas/baz/bip/src/__snapshots__/bop.spec.ts.snap # <<<<<<======== NOTE THIS LINE

✗ yarn workspace foo lint:fix
yarn workspace v1.22.19
yarn run v1.22.19
$ eslint src --fix
(node:42266) ExperimentalWarning: Importing JSON modules is an experimental feature and might change at any time
(Use `node --trace-warnings ...` to show where the warning was created)
=============

WARNING: You are currently running a version of TypeScript which is not officially supported by @typescript-eslint/typescript-estree.

You may find that it works just fine, or you may not.

SUPPORTED TYPESCRIPT VERSIONS: >=4.7.4 <5.6.0

YOUR TYPESCRIPT VERSION: 5.6.2

Please only submit bug reports when using the officially supported version.

=============
✨  Done in 1.06s.
✨  Done in 1.21s.

config
import presets from<EMAIL_ADDRESS>
// pull in the list of packages that are included in the base layer for lambdas
import layerPackageJson from './layers/node-dependencies/nodejs/package.json' assert { type: 'json' };
const packages = Object.keys(layerPackageJson.dependencies);

export default [
  ...presets,
  {
    rules: {
      'import/no-extraneous-dependencies': 'warn',
      // overwrite extended base eslintrc sort-keys warning
      'sort-keys': [
        'error',
        'asc',
        { caseSensitive: true, natural: false, minKeys: 6 },
      ],
    },
    settings: {
      'import/core-modules': [2, ...packages],
    },
  },
];

base config
import js from '@eslint/js';
import ts from '@typescript-eslint/eslint-plugin';
import tsParser from '@typescript-eslint/parser';
import prettier from 'eslint-config-prettier';
import importPlugin from 'eslint-plugin-import';
import jest from 'eslint-plugin-jest';
import simpleImportSort from 'eslint-plugin-simple-import-sort';
import globals from 'globals';

/** @type {import('eslint').Linter.Config[]} */

export default [
  {
    files: ['**/*.{js,ts,jsx,tsx}'],
    languageOptions: {
      parser: tsParser,
      globals: {
        ...globals.node,
      },
    },
    plugins: {
      js,
      ts,
      '@typescript-eslint': ts,
      simpleImportSort,
      import: importPlugin,
    },
    ignores: ['.eslintrc.*', 'jest.config.js', 'cdk.out/**/*'],
    rules: {
      ...js.configs['recommended'].rules,

      ...ts.configs['recommended'].rules,

      '@typescript-eslint/no-unused-vars': 'error',
      '@typescript-eslint/no-explicit-any': 'error',

      'import/no-duplicates': 'error',
      'import/no-cycle': 'error',
      'import/no-extraneous-dependencies': 'error',

      // https://eslint.org/docs/latest/rules/no-redeclare#handled_by_typescript
      'no-redeclare': 'off',

      'simpleImportSort/imports': [
        'error',
        {
          // adds default custom grouping for @nutrien packages
          // see https://github.com/lydell/eslint-plugin-simple-import-sort#custom-grouping
          groups: [['^\\u0000', '^@?\\w'], ['^@nutrien'], ['^', '^\\.']],
        },
      ],
      'simpleImportSort/exports': 'error',
    },
  },
  {
    files: ['**/jest.setup.ts', '**/__tests__/**/*', '**/*.{spec,test}.*'],
    plugins: {
      jest,
    },
    languageOptions: {
      globals: {
        ...globals.jest,
      },
    },
    rules: {
      ...jest.configs['recommended'].rules,

      '@typescript-eslint/explicit-function-return-type': 'off',
      '@typescript-eslint/no-empty-function': 'off',
      '@typescript-eslint/no-explicit-any': 'off',
      '@typescript-eslint/no-non-null-assertion': 'off',

      'jest/no-alias-methods': 'warn',
      'jest/prefer-to-contain': 'warn',
      'jest/prefer-to-have-length': 'warn',
    },
  },
  // should be the last config in order to override preceding rules
  prettier,
];

the file is named bop.spec.ts.snap., with a trailing dot?
Also, you're using eslint 9, but can you confirm the version of the import plugin you're using?

the file is named bop.spec.ts.snap.,

No, that's a typo from adding spaces after the end of the word. Two spaces and OSX inserts a period.

Also, you're using eslint 9, but can you confirm the version of the import plugin you're using?

"eslint-plugin-import": "^2.27.5", which is resolving to 2.29.1 in the node modules folder.
Looks like that's out of date. I'll update and see if it fixes it.

Indeed, that version doesn't support eslint 9 and you shouldn't be using them together :-)

I set "eslint-plugin-import": "2.31.0", in my root package.json and the error is still happening.
✗ cat node_modules/eslint-plugin-import/package.json | grep version
  "version": "2.31.0",

However I'm not convinced that ESLint is actually using 2.31.0 when running the command. Is there some way to check the versions that ESLint is using for plugins when it runs? No --verbose and --debug isn't giving me that information.

I'm trying to just ignore any __snapshot__ folder instead of having to sort all this out. But when I add
ignores: ['**/__snapshot__/']

to my root eslint config
I get an error You are linting "src", but all of the files matching the glob pattern "src" are ignored. I take out that ignores block and the problem goes away. But the docs say:

If you want to recursively ignore all directories named .config, you need to use **/.config/, as in this example:

// eslint.config.js
export default [
    {
        ignores: ["**/.config/"]
    }
];

are the underscores in the path screwing things up? I tried escaping them and then Prettier removed the escapes.

You don't need to escape underscores, no.

You don't need to escape underscores, no.

I got it sorted, see Edit block in that comment.

Awesome! What's the mismatch you see in the docs? Let's fix it :-)

The docs say this should work (the full quote is up above):
 ignores: ["**/.config/"]
but when I add
ignores: ['**/__snapshot__/']
I get an error about src being ignored. I can fix it with
ignores: ['**/*.snap'],

to ignore all of the files with .snap extension it works fine.


Install Error

Using python 3.6 in conda environment. Getting the following error
conda install -c maciejkula -c soumith spotlight=0.1.2
Fetching package metadata .............
PackageNotFoundError: Package missing in current osx-64 channels:

spotlight 0.1.2*

Sorry about that. I haven't pushed 0.1.2 to the conda channels yet. You can
clone this repository or install 0.1.1 from Conda. I'll get the new version
uploaded on the next couple of days.
On 18 Sep 2017 22:46, "Naveed Ahmad"<EMAIL_ADDRESS>wrote:

Using python 3.6 in conda environment. Getting the following error
conda install -c maciejkula -c soumith spotlight=0.1.2
Fetching package metadata .............
PackageNotFoundError: Package missing in current osx-64 channels:

spotlight 0.1.2*

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/maciejkula/spotlight/issues/61, or mute the thread
https://github.com/notifications/unsubscribe-auth/ACSCAwMcQM_2LSV_HlXIh2SK5oNIzxMBks5sjuSYgaJpZM4Pbkqv
.

Hi Maciej,
Thanks, that worked!
Thanks for the package. I am interested in prototyping news recommendation,
using webstream and content data. Would youtube, or lstm models be a good
choice, and have you tested using any open source web data?
Regards
On Mon, Sep 18, 2017 at 6:32 PM, Maciej Kula<EMAIL_ADDRESS>wrote:

Sorry about that. I haven't pushed 0.1.2 to the conda channels yet. You can
clone this repository or install 0.1.1 from Conda. I'll get the new version
uploaded on the next couple of days.
On 18 Sep 2017 22:46, "Naveed Ahmad"<EMAIL_ADDRESS>wrote:

Using python 3.6 in conda environment. Getting the following error
conda install -c maciejkula -c soumith spotlight=0.1.2
Fetching package metadata .............
PackageNotFoundError: Package missing in current osx-64 channels:

spotlight 0.1.2*

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/maciejkula/spotlight/issues/61, or mute the thread
<https://github.com/notifications/unsubscribe-auth/ACSCAwMcQM_2LSV_
HlXIh2SK5oNIzxMBks5sjuSYgaJpZM4Pbkqv>
.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/maciejkula/spotlight/issues/61#issuecomment-330374946,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABMlXRVtt1BuuRVthviTldDdf0jIWUBoks5sju-QgaJpZM4Pbkqv
.

The LSTM models should be a good fit.
I'm still working on a nice way of integrating content data into the models; I imagine that would be crucial for news recommendations.

Hi Again,
I am going through the code and understand how its implemented.
In sequence/representations.py line 193
self.lstm = nn.LSTM(batch_first=True,
input_size=embedding_dim,
hidden_size=embedding_dim)
I do not understand how the LSTM is being used. It is not referenced in the
forward function. Also CNNNet, LSTMNet, PoolNet have the same
implementation for forward methods.
Is this code still work in progress, or am I missing something?
Regards
On Wed, Sep 20, 2017 at 2:11 PM, Maciej Kula<EMAIL_ADDRESS>wrote:

The LSTM models should be a good fit.
I'm still working on a nice way of integrating content data into the
models; I imagine that would be crucial for news recommendations.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/maciejkula/spotlight/issues/61#issuecomment-330935050,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABMlXSekgsoW7byZTgQ4eh7cioHPQj38ks5skVVSgaJpZM4Pbkqv
.

The key differences are in the user_representation functions. These feed into the forward methods.


Is this an ESM-first module now?

When bumping from 10.3.0 to 10.4.4 I encountered the following issue when building my project:
> Build error occurred
Error [ERR_REQUIRE_ESM]: Must use import to load ES Module: /home/runner/work/monorepo/monorepo/website/node_modules/d3-array/src/index.js
require() of ES modules is not supported.
require() of /home/runner/work/monorepo/monorepo/website/node_modules/d3-array/src/index.js from /home/runner/work/monorepo/monorepo/website/node_modules/reaviz/dist/index.cjs.js is an ES module file as it is a .js file whose nearest parent package.json contains "type": "module" which defines all .js files in that package scope as ES modules.
Instead rename index.js to end in .cjs, change the requiring code to use import(), or remove "type": "module" from /home/runner/work/monorepo/monorepo/website/node_modules/d3-array/package.json.

Is this an ESM-first module now?
In my opinion, this should be clearly stated in the documentation.
Also, since this breaks backward compatibility (given the current JS ecosystem), I think this should warrant a major version change; not a minor one.

@botre - sorry to hear about the issue. let me provide some feedback below.

Is this an ESM-first module now?
It shouldn't be. In the latest release, I upgraded the d3- dependencies which are ESM now, however, these are bundled with the project so they shouldn't  be causing this issue.

In my opinion, this should be clearly stated in the documentation.
100% - I did not realize this was happening. If it was I would have documented and bumped major version.

I'm running the project in 2 different CRA applications and not seeing this error. Did you manually include those deps? Can you try deleteing your yarn/npm lock and reinstalling?

I think CRA supports ESM and transpilation out of the box (unlike Next.js and other tools).
Deleting NPM lock and re-installing did not solve the issue :(

I have created a minimal reproduction for this issue, see this repo: https://github.com/botre/reaviz-next-repro

Thank you - Im digging into this - will let you know tomorrow how we can proceed

Let me know if there is anything I can help with.

OK - I bumped the version to a major so others are aware. I don't know the fix yet though. REAVIZ does have a CJS version but I don't know why the d3-* packages are being difficult. I will keep digging in and let me know if you find anything.

Hey! Just found this thread, same issue. Has there been a resolution? I am up for a bug hunt if help is required

@jazznerd206  No resolution yet - any help would be greatly appreciated!

@amcdnl
yeah buddy. i'll give it a shot.

I don't wanna get too far ahead of myself, but d3 might have solved the problem for us.
The specific dep that's throwing the error for me is d3-array, and according to your package.json, you have v3.0.4, and there is a new version, 3.1.1 available as of 1 month ago. According to the changelog, they have updated it to es6, so import should actually work now.
I'll report back after some testing.

Ya - This started when d3-* made their packages ESM and since its a dep of reaviz if your app isn't setup to handle esm then the error happens.

@amcdnl Hey, so it turns out that was exactly my problem, not a problem with reaviz!!
my fix was as follows:
webpack.config.js
{         test: /\b.json\b/,         loader: 'json-loader',  },
sorry for conflating my mistake with your package!!

Great news - glad you were able to resolve it. Let the group know if you find any of suggestions for this!

reactviz is also breaking on latest Next (12.0.7) which is supposed to support ESM packages by default.
In the end managed to get it working with dynamic imports, probably not ideal but at least it's a workaround 🤷🏼
import dynamic from "next/dynamic";

const Component = () => {
  const [CalendarHeatmap, setCalendarHeatmap] = useState<any>(null);
  useEffect(() => {
    // mod.* can be anything you would normally import like BarChart, LineChart etc
    const DynamicCalendarHeatmap = dynamic(() => import('reaviz' as any).then(mod => mod.CalendarHeatmap))

    if(DynamicCalendarHeatmap) {
      setCalendarHeatmap(DynamicCalendarHeatmap)
    }
  }, [])

  return (
    <div>
      {CalendarHeatmap && <CalendarHeatmap data={calendarData} height={300} width="auto"/>}
    </div>
  );
}

"reaviz": "^11.1.2",
"next": "^12.0.7",
Tried to debug the issue but no success on my end either. Here is what I tried:

use next-transpile-modules in next.config.js- still broken
removed cjs from rollup (including commonJs plugin) - still broken
use @jazznerd206 solution - still broken
bumped d3-array to 3.1.1 - still broken
for science went with experimental.esmExternals = true in next.config.js even though it's true by default since 12.x.x (may as well  try that at this point lol) - still broken
change rollup-plugin-commonjs to @rollup/plugin-commonjs - still broken
downgraded to<EMAIL_ADDRESS>and tried out with and without experimental.esmExternals = true - still broken
tried urlImports with https://cdn.skypack.dev/reaviz - still broken

At that point the only thing that left besides dynamic import in my mind was to dig in the source maps but didn't bother once I got it working

@sbsrnt - Thanks for the detailed report. I'm open to suggestions on ways to address this - I'm not sure what the best path is though.

https://github.com/vercel/next.js/issues/33901#issuecomment-1027842861

@dpranav007 - Thanks for the ping. The comment in that ticket is very helpful. I need to research an alternative build configurations and in the meantime if u find anything relevant post in this thread.

In the end managed to get it working with dynamic imports, probably not ideal but at least it's a workaround 🤷🏼

This is still an issue for with next version 12.1.0 (and reavis 13.0.4) but I've got a much simpler work around for anyone coming across this:
import dynamic from "next/dynamic";
const BarChart = dynamic(() => import("reaviz").then((mod) => mod.BarChart), {
  ssr: false,
});

Next docs. The SSR :false option seems to fix this issue (at the obvious cost of not rendering on the server)

Hi I've tried using @zischwartz solution but it's not working when I try to import LinearXAxis and LinearYAxis. Below is my code.
import React from 'react';
import { Card, CardContent, CardHeader } from '@mui/material';
import dynamic from 'next/dynamic';

const BarChart = dynamic(() => import('reaviz').then(mod => mod.BarChart), {ssr: false});
const BarSeries = dynamic(() => import('reaviz').then(mod => mod.BarSeries), {ssr: false});
const LinearXAxis = dynamic(() => import('reaviz').then(mod => mod.LinearXAxis), {ssr: false});
const LinearYAxis = dynamic(() => import('reaviz').then(mod => mod.LinearYAxis), {ssr: false});

const data = [
    {key: "Browser Session Hijacking - T1185", data: 5},
    {key: "Build Image on Host - T1612", data: 5},
    {key: "	Access Token Manipulation - T1134", data: 10},
    {key: "Boot or Logon Autostart Execution - T1235", data: 10},
    {key: "Bypass User Account Control - T1070", data: 20},
    {key: "Command-Line Interface - T1059", data: 50},
    {key: "Malicious File - T1204", data: 100},
    {key: "PowerShell - T1059", data: 150},
    {key: "Remote Desktop Protocol - T1021", data: 200},
    {key: "Data Encrypted for Impact - T1486", data: 300},
]

export default function MitreBarChart({ height, width }) {
    return (
        <Card style={{height: height, width: width}}>
            <CardHeader title="Most Detected Types of Threats"/>
            <CardContent style={{height: "100%", width:"100%", padding: "10px 20px 0 105px"}}>
                <BarChart
                    data={data}
                    xAxis={<LinearXAxis type="value"/>}
                    yAxis={<LinearYAxis type="category"/>}
                    series={<BarSeries layout="horizontal"/>}
                />
            </CardContent>
        </Card>
    )
}

This is the error:

I'm happy to implement any suggestion someone might have, I just don't know the best way to fix this. I've also not hit it but I'm not doing NEXT/etc.

same issue, not work in cra with ssr:
module.exports = require("d3-time-format");

Error [ERR_REQUIRE_ESM]: require() of ES Module .../node_modules/d3-time-format/src/index.js from .../build/server.js not supported.
[1] Instead change the require of index.js in .../build/server.js to a dynamic import() which is available in all CommonJS modules.

Please give this a try again!


Adding discord authentication

I'll take a shot at it :-D

You got it!

See pull request. Base functionality completed. Now it's time to get fancy with it!


Syntastic instructions

Added syntastic instructions to readme - these have been tested

Actually this might have gone to the wrong place... trying again...


VC4CL installation error

Hello.
I was able to build VC4CLStdLib, VC4C and VC4CL.
I installed both VC4CLStdLib and VC4C by:
$ sudo dpkg -i /tmp/vc4cl-stdlib.deb
$ sudo dpkg -i /tmp/vc4c.deb

But, when I install VC4CL by
$ sudo dpkg -i /tmp/vc4cl.deb

I get:
$ dependency problems prevent configuration of vc4cl:
$   vc4cl depends on opencl-c-headers; however Package opencl-c-headers is not installed.

This is strange because I've installed it by:
$ sudo apt-get install opencl-headers

Any ideas?
Thanks in advance.

It says requires opencl-c-headers (note the C), these this is a different package than opencl-headers. One is run-time (host-side) the other kernel-side headers.

Thanks for the answer.
Do you know how to install it?
I've tried
sudo apt install opencl-c-headers
``'
But without success.

Maybe ubuntu 16.04 does not support opencl-c-headers package?

Debian provides them, so they should be in the repositories... If not, you can manually download them from the link I provided.

I've installed opencl-c-headers from the link but it currupts the installation of opencl-headers. I'm really confused...
Also, clinfo returns 0 platforms.
Can't you provide a full installation guide?

For Raspbian, the full installation guide is as follows:

Install vc4cl-stdlib package
Install dependencies clang-3.9, llvm-3.9, llvm-3.9-dev and vc4c package
Install dependencies opencl-c-headers, ocl-icd-opencl-dev and vc4cl package

At least that is the guide for Raspbian. For any other Debian it should be similar.
clinfo will return 0 platforms if you did not yet install the vc4cl package successfully and have no other OpenCL run-time installed.

Also note, for other OSes (other than Raspbian), you will need to make sure, the required kernel modules and the Raspberry Pi userland libraries are available.
See also #53 for how to port to other distributions

Thanks for all the support.
I'll try to install Raspbian instead of Ubuntu and see how it goes.
Just to check. To install VC4C/VC4CL-StdLib/VC4CL I should do
cmake ..
sudo make install

in each build directory, right?

Well, you should build without root rights and only install with root rights. But otherwise yes. The order is important though, you need to install VC4CLStdLib before building VC4C!

Installing raspbian do the trick, thank you very much!
Now, I will try to link VC4CL with ROS.
Any suggestion?

That depends on what ROS is?

ROS = Robot Operating System.
My main objective is to optimize the processing time of a Visual Odometry algorithm that runs over ROS.

Note that ROS compiles using Cmake!

If the ROS supports OpenCL and supports using the OpenCL ICD loader, then it should work out-of-the-box.

I'm not sure... But I'll give it a try.
Thanks for all the support.
I think you can close the issue.


So no need to mangle with internals using this SDK for non-EDU?

Hey! Awesome project, i just got g1 air for my locomotion project. But i am puzzled does this mean that we don't need to have any firmware tweaks to make it work unlike official one?

Correct... the pay to play limitation of EDU dog using LOW level is artificial. Air, Pro, and MAX can all use Low level commands, not just EDU. No firmware tweaks required.

If this helped you to save money feel free to donate to help source more robot dogs :) https://www.buymeacoffee.com/bin4ry


WM: Add support for Kwm on macOS

Description
This PR adds support for Kwm.

Change the ps line to this:
ps -e | grep -F -q "kwm" && wm="Kwm"

Otherwise LGTM :+1:

Done.

Thanks!

They always seem to fail on my PRs. Would you happen to know why?

Travis runs shellcheck and travis errors if shellcheck doesn't like anything.

So I made the change you suggested without actually testing it.
[[ "$(ps -e | grep '[k]wm')" ]] && wm="Kwm" -> ps -e | grep -F -q "kwm" && wm="Kwm"
This change doesn't work as intended.

I thought you would've tested it before changing the PR.
What doesn't work exactly?

My apologies.
The line ps -e | grep -F -q "kwm" && wm="Kwm" sets the window manager to "Kwm" even if it isn't actually running as a process.
This happens because the grep process always contains the "kwm" string when searching for it. grep  "[k]wm" prevents that process from showing up in the output.
https://unix.stackexchange.com/questions/74185/how-can-i-prevent-grep-from-showing-up-in-ps-results?noredirect=1&lq=1
ps -e | grep -q '[k]wm' && wm="Kwm"  is the proper fix.

That was my fault, don't worry.
Thanks for finding the correct fix. :)
Feel free to open another PR and I'll merge it. :+1:

Done.


fix(nginx-ingress): move nginx-ingress configs from ./helmfile.d/nginx-ingress.yaml to their own files in ./config/default

Fixes #1653

No diff, all good :)


All source code files must begin with the Apache License Header

Description
Subtask of #1909
Describe the solution you'd like
Add apache header to every single source code files
header can be found from all apache project, like https://github.com/apache/servicecomb-java-chassis/blob/master/core/src/main/java/org/apache/servicecomb/core/bootstrap/SCBBootstrap.java
Has the Feature been Requested Before?
No

ok, i'll add it

ok, i'll add it

finished