open_mfdataset() significantly slower on 0.9.1 vs. 0.8.2

I noticed a big speed discrepancy between xarray versions 0.8.2 and 0.9.1 when using open_mfdataset() on a dataset ~ 1.2 GB in size, consisting of 3 files and using netcdf4 as the engine. 0.8.2 was run first, so this is probably not a disk caching issue.
Test
import xarray as xr
import time

start_time = time.time()
ds0 = xr.open_mfdataset('./*.nc')
print("--- %s seconds ---" % (time.time() - start_time))

Result
xarray==0.8.2, dask==0.11.1, netcdf4==1.2.4
--- 0.736030101776 seconds ---

xarray==0.9.1, dask==0.13.0, netcdf4==1.2.4
--- 52.2800869942 seconds ---

I've also noticed that we have a bottleneck here.
@shoyer - any idea what we changed that could impact this? Could this be coming from a change upstream in dask?

Wow, that is pretty bad.
Try setting compat='broadcast_equals' in the open_mfdataset call, to restore the default value of that parameter prior v0.9.
If that doesn't help, try downgrading dask to see if it's responsible. Profiling results from %prun in IPython would also be helpful at tracking down the culprit.

This is what I'm seeing for my %prun profiling:
   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
      204   19.783    0.097   19.783    0.097 {method 'acquire' of '_thread.lock' objects}
89208/51003    2.524    0.000    5.553    0.000 indexing.py:361(shape)
        1    1.359    1.359   37.876   37.876 <string>:1(<module>)
71379/53550    1.242    0.000    3.266    0.000 utils.py:412(shape)
   538295    0.929    0.000    1.317    0.000 {built-in method builtins.isinstance}
24674/13920    0.836    0.000    4.139    0.000 _collections_abc.py:756(update)
        9    0.788    0.088    0.803    0.089 netCDF4_.py:178(_open_netcdf4_group)```

Weren't there some recent changes to the thread lock related to dask distributed?

Hmm. It might be interesting to try lock=threading.Lock() to revert to the old version of the thread lock as well.

My 2cents - I've found that with big files any %prun tends to show method 'acquire' of '_thread.lock' as one of the highest time but it's not necessarily indicative of where the perf issue comes from because it's effectively just waiting for IO which is always slow. One thing that helps get a better profile is setting dask backend to the non-parallel sync option which gives cleaner profiles.

One thing that helps get a better profile is setting dask backend to the non-parallel sync option which gives cleaner profiles.

Indeed, this is highly recommended, see http://dask.pydata.org/en/latest/faq.html

I just tried this on a few different datasets. Comparing python 2.7, xarray 0.7.2, dask 0.7.1 (an old environment I had on hand) with python 2.7, xarray 0.9.1-28-g1cad803, dask 0.13.0 (my current "production" environment), I could not reproduce. The up-to-date stack was faster by a factor of < 2.

Data: Five files that are approximately 450 MB each.
venv1
dask                      0.13.0                   py27_0    conda-forge
xarray                    0.8.2                    py27_0    conda-forge
1.51642394066 seconds to load using open_mfdataset
venv2:
dask                      0.13.0                   py27_0    conda-forge
xarray                    0.9.1                    py27_0    conda-forge
279.011202097 seconds to load using open_mfdataset
I ran the same code in the OP on two conda envs with the same version of dask but two different versions of xarray. There was a significant difference in load time between the two conda envs.
I've posted the data on my work site if anyone wants to double check: https://marine.rutgers.edu/~michaesm/netcdf/data/

There is definitely something funky with these datasets that is causing xarray to go very slow.
This is fast:
>>> %time dsets = [xr.open_dataset(fname) for fname in glob('*.nc')]
CPU times: user 1.1 s, sys: 664 ms, total: 1.76 s
Wall time: 1.78 s

But even just trying to print the repr is slow
>>> %time print(dsets[0])
CPU times: user 3.66 s, sys: 3.49 s, total: 7.15 s
Wall time: 7.28 s

Maybe some of this has to do with the change at 0.9.0 to allowing index-less dimensions (i.e. coordinates are optional). All of these datasets have such a dimension, e.g.
<xarray.Dataset>
Dimensions:                                     (obs: 7247697)
Coordinates:
    lon                                         (obs) float64 -124.3 -124.3 ...
    lat                                         (obs) float64 44.64 44.64 ...
    time                                        (obs) datetime64[ns] 2014-11-10T00:00:00.011253 ...
Dimensions without coordinates: obs
Data variables:
    oxy_calphase                                (obs) float64 3.293e+04 ...
    quality_flag                                (obs) |S2 'ok' 'ok' 'ok' ...
    ctdbp_no_seawater_conductivity_qc_executed  (obs) uint8 29 29 29 29 29 ...
...

And the length of obs is different in each dataset.
>>> for myds in dsets:
            print(myds.dims)
Frozen(SortedKeysDict({u'obs': 7537613}))
Frozen(SortedKeysDict({u'obs': 7247697}))
Frozen(SortedKeysDict({u'obs': 7497680}))
Frozen(SortedKeysDict({u'obs': 7661468}))
Frozen(SortedKeysDict({u'obs': 5750197}))

Looks like the issue might be that xarray 0.9.1 is decoding all timestamps on load.
xarray==0.9.1, dask==0.13.0
da.set_options(get=da.async.get_sync)
%prun -l 10 ds = xr.open_mfdataset('./*.nc')

         167305 function calls (160352 primitive calls) in 59.688 seconds

   Ordered by: internal time
   List reduced from 625 to 10 due to restriction <10>

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
       18   57.057    3.170   57.057    3.170 {pandas.tslib.array_to_timedelta64}
       39    0.860    0.022    0.863    0.022 {operator.getitem}
       48    0.402    0.008    0.402    0.008 {numpy.core.multiarray.arange}
4341/2463    0.257    0.000    0.273    0.000 utils.py:412(shape)
       88    0.245    0.003    0.245    0.003 {pandas.algos.ensure_object}
       48    0.158    0.003    0.561    0.012 indexing.py:318(_index_indexer_1d)
    36/18    0.135    0.004   57.509    3.195 timedeltas.py:150(_convert_listlike)
       18    0.126    0.007    0.130    0.007 nanops.py:815(_checked_add_with_arr)
       51    0.070    0.001    0.070    0.001 {method 'astype' of 'numpy.ndarray' objects}
  676/475    0.047    0.000   58.853    0.124 {numpy.core.multiarray.array}

pandas.tslib.array_to_timedelta64 appears to be the most expensive item on the list, and isn't being run when using xarray 0.8.2.
xarray==0.8.2, dask==0.13.0
da.set_options(get=da.async.get_sync)
%prun -l 10 ds = xr.open_mfdataset('./*.nc')

         140668 function calls (136769 primitive calls) in 0.766 seconds

   Ordered by: internal time
   List reduced from 621 to 10 due to restriction <10>

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
2571/1800    0.178    0.000    0.184    0.000 utils.py:387(shape)
       18    0.174    0.010    0.174    0.010 {numpy.core.multiarray.arange}
       16    0.079    0.005    0.079    0.005 {numpy.core.multiarray.concatenate}
  483/420    0.077    0.000    0.125    0.000 {numpy.core.multiarray.array}
       15    0.054    0.004    0.197    0.013 indexing.py:259(_index_indexer_1d)
        3    0.041    0.014    0.043    0.014 netCDF4_.py:181(__init__)
      105    0.013    0.000    0.057    0.001 netCDF4_.py:196(open_store_variable)
       15    0.012    0.001    0.013    0.001 {operator.getitem}
2715/1665    0.007    0.000    0.178    0.000 indexing.py:343(shape)
     5971    0.006    0.000    0.006    0.000 collections.py:71(__setitem__)

The version of dask is held constant in each test.

@rabernat This data is computed on demand from the OOI (http://oceanobservatories.org/cyberinfrastructure-technology/). Datasets can be massive and so they seem to be split up in ~500 MB files when data gets too big. That is why obs changes for each file. Would having obs be consistent across all files potentially make open_mfdataset faster?

My understanding is that you are concatenating across the variable obs, so no, it wouldn't make sense to have obs be the same in all the datasets.
My tests showed that it's not necessarily the concat step that is slowing this down. Your profiling suggest that it's a netcdf datetime decoding issue.
I wonder if @shoyer or @jhamman have any ideas about how to improve performance here.

@friedrichknuth Did you try a tests with the most recent version decode_times=True/False ?

decode_times=False significantly reduces read time, but the proportional performance discrepancy between xarray 0.8.2 and 0.9.1 remains the same.

@friedrichknuth, any chance you can take a look at this with the latest v0.10 release candidate?

Looks like it has been resolved! Tested with the latest pre-release v0.10.0rc2 on the dataset linked by najascutellatus above. https://marine.rutgers.edu/~michaesm/netcdf/data/
da.set_options(get=da.async.get_sync)
%prun -l 10 ds = xr.open_mfdataset('./*.nc')

xarray==0.10.0rc2-1-g8267fdb
dask==0.15.4

         241253 function calls (229881 primitive calls) in 98.123 seconds

   Ordered by: internal time
   List reduced from 659 to 10 due to restriction <10>

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
       30   87.527    2.918   87.527    2.918 {pandas._libs.tslib.array_to_timedelta64}
       65    7.055    0.109    7.059    0.109 {operator.getitem}
       80    0.799    0.010    0.799    0.010 {numpy.core.multiarray.arange}
7895/4420    0.502    0.000    0.524    0.000 utils.py:412(shape)
       68    0.442    0.007    0.442    0.007 {pandas._libs.algos.ensure_object}
       80    0.350    0.004    1.150    0.014 indexing.py:318(_index_indexer_1d)
    60/30    0.296    0.005   88.407    2.947 timedeltas.py:158(_convert_listlike)
       30    0.284    0.009    0.298    0.010 algorithms.py:719(checked_add_with_arr)
      123    0.140    0.001    0.140    0.001 {method 'astype' of 'numpy.ndarray' objects}
 1049/719    0.096    0.000   96.513    0.134 {numpy.core.multiarray.array}

xarray==0.9.1
dask==0.13.0
241253 function calls (229881 primitive calls) in 98.123 seconds


Ability to modify the policy xml file via environment specific publisher configuration

Please describe the feature.
I am not sure if this is supported already but how do we modify a value inside a policy.xml file based on environment specific publisher configuration. In the sample artifacts, the global policy.xml has 'environment' attribute which should be replaced during runtime as part of publishing process but I didn't see a reference in configuration.prod.yaml for the same. Appreciate your help

For this use case we recommend that you utilize namevalue pairs and simply override them across environments.


CBL-273 : Add an internal API for restarting concurrent executor

The concurrent executor is shared so it is needed to be restarted when tests stop it. The issue didn’t happen on CBL Android because somehow the CBL Android tests get executed in an order that doesn’t cause the problem.

Added AbstractExecutorService’s restartConcurrentExecutor() method that will call the restart() method of the ConcurrentExecutor.

Currently the only way to create concurrent executor is through CouchbaseLite.executionService().getConcurrentExecutor(). We could to @before but the followings need to be done.

Make AbstractExecutor.ConcurrentExecutor visible for creation.
Calls new ConcurrentExecutor(getThreadPoolExecutor()) directly to create insteads of CouchbaseLite.executionService().getConcurrentExecutor()

What do you think?

Look like I read your comment wrong. To create a new ExecutionService every time, we need to add another method (probably in CouchbaseLite) to deal with creating JavaExecutionService or AndroidExecutionService.

well.. I think that the AndroidExecutionService is already public, so that's half the problem.  The @Before would just create a new ExecutionService for each test.  That seems ok.
The problem is creating the right one for the right platform.   I don't know how hard that would be.  Could it go in BaseTest and this just depend on that?
This really does seem like something worth considering.
Still, it is a bug that is distracting you from your work.  I think your solution is reasonable.  Use it, if you want.

OK, I will not change the PR.


The wc request is duplicated

Store the received request.id in an array in memory. If the request id is already in the array, do not process it further.

This bug already fixed in Android dev version i1.5.0


What hardware is everyone using for this?

Just curious what hardware works best with this, that people have had success with.

I am using a Beaglebone for running node-red and homebridge. Node-red is running the automation logic and providing the API endpoints for the homebridge-http-securitysystem.

Ha! I should have been more specific. I meant alarm system hardware. Like the keypad, panel, the sensors... etc


Add integration tests that validates if craco is compatible with the latest version of CRA

Add integration tests that validates if craco is compatible with the latest version of CRA

I've written a test for craco-antd, so that might be helpful as a reference: https://github.com/FormAPI/craco-antd/blob/master/lib/craco-antd.test.js


licence import

Hello,
I pasted the the license file in the config directory, where there the cache, log, tmp directories are.
when starting the docker, the license is not taken into consideration
HW: Linux 1a8f23b1c163 4.19.107-Unraid #1 SMP Sun Mar 8 14:34:03 CDT 2020 x86_64 GNU/Linux STORAGE: btrfs [/] @ 38 GB | fuse.shfs [/storage] @ 4.0 TB | fuse.shfs [/watch] @ 3.9 TB | fuse.shfs [/output] @ 3.9 TB | fuse.shfs [/config] @ 179 GB | btrfs [/etc/resolv.conf] @ 38 GB | btrfs [/etc/hostname] @ 38 GB | btrfs [/etc/hosts] @ 38 GB DATA: /config Package: DOCKER License: UNREGISTERED
the script is invoiced, and I get a license error
`[amc] License Error: UNREGISTERED
[amc] ________________________________________________
[amc] / Please purchase a FileBot License: https://www.filebot.net/purchase.html#docker /`

I found the solution, had to rename the file that is has no special signs and no capitals

Could you share the name of the filename that was not working ?  Maybe there is something that can be enhanced/fixed to avoid these problems.


fix log 0 error in noise adaptive layout

fixed #2644

I believe that this commit introduces a correctness bug. If the gate's error was infinitesimally close to one, g_reliab would be infinitesimally close to zero -- lets say 1e-14, making swap_reliab
-log((1e-14)**3) = 96
a positive number. As the error gets closer to one, -log(swap_reliab) will go to infinity, not -infinity. Line 100 should be
            swap_reliab = -math.log(swap_reliab) if swap_reliab != 0 else math.inf

Currently, using -math.inf actually maps the circuit to broken links rather than away from them.


Change InstaScale Configmap name

Relevant InstaScale PR: https://github.com/project-codeflare/instascale/pull/52
InstaScale always expects a configmap with the name instascale-config to exist in a namespace. Our current configmap name does not align with that -- we should fix it.

As part of this we will also need to update the deployment template to pass in the configmap namespace.

This has become obsolete with #279.


Show Manual (F1) Icon references unknown App on mac

Clicking on the manual icon causes the following debug message on the glade invocation console...
Unable to find application named 'Help Viewer.app'
I was unable to locate any information about where from or how to install the proper viewer so this will work correctly. This is not a show stopper, just a little issue.
Thanks

wxGlade/main.py:
at line 1098:
if wx.Platform == "WXMAC":
os.system(r'open -a Help\ Viewer.app %s' % html_file)
This seems to be where the action is. I don't know if this app used to exist, or whether this was meant to be configurable. Maybe a soft link in the Applications folder to Safari? There is nothing in the App Store called "Help Viewer".

Thanks for reporting. It seems that only HelpViewer is available now, but a quick check did not work with command line arguments.
The better option seems to be to just call open.
Does it work for you if you replace the os.system line with
os.system(r'open "%s"' % html_file) ?

I modified the os call to use safari instead, since that is always available on the mac.
It’s job is opening html files after all.

On Dec 3, 2018, at 10:46 AM, Dietmar Schwertberger<EMAIL_ADDRESS>wrote:
Did you try the modified os.system call?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub https://github.com/wxGlade/wxGlade/issues/350#issuecomment-443822614, or mute the thread https://github.com/notifications/unsubscribe-auth/AMYiRO-gTbc0bklkQKR2XH0HKEwyKXUEks5u1XGFgaJpZM4Y3x9_.

Fixed in 0.9.0b2


rejectVersionIf not utilized for catalog library entry

Describe the issue
Using the experimental resolver nl.littlerobots.vcu.resolver, appears to not utilize the rejectVersionIf from the dependencyUpdates plugin configuration.  I have created a sample project where a rejectVersionIf statement is created, but not utilized by the command versionCatalogUpdate.
Steps to reproduce
With the sample project running ./gradlew clean versionCatalogUpdate --interactive a libs.versions.updates.toml is created with the dependency commons-io version of 20030203.000550.  This version is provided because of incorrect metadata in the repository https://repo.grails.org/core, see settings.gradle in the sample project.  The expected behavior was that the rejectVersionIf statement would be utilized and the version 20030203.000550 would not be chosen.
Can you provide guidance on how to utilize a reject version strategy against a catalog library entry?

From your sample project, you are configuring rejectVersionIf on the dependencyUpdates task. However, that task is not used at all when you switch to the new resolver. You should use a version selection block in stead and compare/filter the versions there:
    versionSelector {
        // return true to select the passed in candidate
        !isNonStable(it.candidate.version)
    }
}```

As an aside, it might just be better to setup a strict rule in your version catalog itself als an alternative.

By the way, thanks for providing a sample project, it really helps me to understand and repro the issue ❤

Appreciate the quick reply!  Glad to see the plugin is now able to do resolution without the version plugin.  One less dependency.
Using the versionSelector from within the versionCatalogUpdate, and am still unable to omit the version 20030203.000550 with task versionCatalogUpdate from the generated catalog.  Here is the versionSelector used
versionSelector {
    it.candidate.version != "20030203.000550"
}

Also updated the sample project accordingly.  Any thoughts as to what is wrong here?

It looks like the selector is not set correctly or too late when Gradle is running. Debugging it looks like the plugin is configured first, and then the version selector method call is processed. By that time the plugin is already configured with a default version selector.
You can set it directly on the task too as a workaround, but I'd prefer to get the method on the extension working if possible.
tasks.named("versionCatalogUpdate").configure {
    versionSelector {
        it.candidate.version != "20030203.000550"
    }
}


deleting bug

same issue occur Both in customer listing page and vendor listing page
1)In customer listing page if customer has opening balance ,user can't delete that customer.
But here when we delete the customer who has opening balance,first it showing requested URL not found
But when we refresh, the customer has deleted already.

2)in vendor listing page if vendor has opening balance user can't delete that vendor

ERROR SOLVED


Chrisda to Master

Docs Build status updates of commit d75be81:
:clock10: Full build: the average full build time is 17 min(s) 42 sec(s), this is based on the last 1000 build(s) for this repository.

Docs Build status updates of commit d75be81:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

exchange/docs-conceptual/connect-to-exchange-online-powershell.md
:white_check_mark:Succeeded
View (exchange-ps)

exchange/docs-conceptual/connect-to-exchange-online-protection-powershell.md
:white_check_mark:Succeeded
View (exchange-ps)

exchange/docs-conceptual/connect-to-scc-powershell.md
:white_check_mark:Succeeded
View (exchange-ps)

exchange/docs-conceptual/exchange-online-powershell-v2.md
:white_check_mark:Succeeded
View (exchange-ps)

exchange/docs-conceptual/mfa-connect-to-exchange-online-powershell.md
:white_check_mark:Succeeded
View (exchange-ps)

exchange/docs-conceptual/mfa-connect-to-scc-powershell.md
:white_check_mark:Succeeded
View (exchange-ps)

exchange/exchange-ps/exchange/Connect-ExchangeOnline.md
:white_check_mark:Succeeded
View (exchange-ps)

exchange/exchange-ps/exchange/Connect-IPPSSession.md
:white_check_mark:Succeeded
View (exchange-ps)

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching in the Docs contributor and Admin GuideSee the frequently asked questionsPost your question in the Docs support channel


Running as a default terminal causes app to create terminal window

Windows Terminal version (or Windows build number)
10.0.22000.0, 1.11.2921.0
Other Software
CoolerMaster MasterPlus (I couldn't find any version string :( but it's a 2-3 week old fresh install)
Steps to reproduce
Make Windows Terminal the default terminal.
Install CoolerMaster MasterPlus (I don't think it's a requirement to have any of their hardware to use it) and on every reboot there's a service called MPService that starts a process with a console host attached:

Then a windows terminal window appears:

And every time I close it (obviously the process ends) the service restarts the program (so the only way to get rid of this is to stop the service).
Expected Behavior
Expected behavior should be the same when conhost was the default console terminal (no window creation, the program ran in the background)
Actual Behavior
The terminal window appears every time this service runs the said program instead of having it in the background somehow.

Same issue using Syncthing, their doc recommends to create a startup task with options to disable the console: "C:\Program Files\Syncthing\syncthing.exe" -no-console -no-browser but on Windows 11 using Windows Terminal as default makes the console window visible instead of hiding it (works perfectly with conhost as default).

I have the same issue with more apps, eg Qt Maintenace Tool for windows opens a visible terminal, I let know about other apps when I discover which one opens these terminals.

Ok, after a few weeks of use I can tell that the Qt Maintenance tool is only one application that opens a visible terminal window, I'm pretty sure that wterm behaves correctly and this will be Qt Maintenance Tool problem. Personally, I have not observed any wrong behavior of the wterm. 😁

There still can be a problem that is described in this issue, I have not tested it with CoolerMaster MasterPlus like is described in the first post, but I have tested with dozens of applications that I'm using.

I just realized the issue in my case is not present anymore. I don't know at what version it was fixed because I had conhost as default and somehow it automatically changed it to windows terminal without me knowing.

Okay, I can confirm that this is seemingly fixed for CoolerMaster. Impossible to say if it was due to an OS-side defterm fix, a Terminal-side fix, or a CoolerMaster-side one.
The ShowWindow thing should have been fixed by #12515
Anything else that's in this bucket seems like it's probably a variant of /dup #12154. I'm gonna keep using that one to track it, since that's a little more narrowed down to the root cause.
Thanks all!


add license title

as dicussed in https://github.com/librosa/librosa/pull/382

Coverage remained the same at 97.917% when pulling efbb1cb423ce15579b180bfe9b5d4f7951d4975a on waldyrious:patch-1 into 13cb7004bfdbfd7e76a4f4a21af8162b97ea2ecb on bmcfee:master.


Add Coveralls

Changes Unknown when pulling 1146bd92e02923eca5ce9cdff4dd302d4dd1f9f1 on coveralls into ** on master**.

Changes Unknown when pulling ad814cd4bfdbc2a351b98cf29f7a2d90598d64ab on coveralls into ** on master**.


Can someone find the fault in this small piece of code?

So what i'm trying to do is take a sample tweet, and follow the recent 20 people who retweeted that tweet
import tweepy
consumer_key = 'XXXXXXXXXXXXXXXXXXXXXX'
consumer_secret = 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'
access_token = '624837175-XXXXXXXXXXXXXXXXXXX'
access_token_secret = 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'
auth = tweepy.OAuthHandler(consumer_key, consumer_secret)
auth.set_access_token(access_token, access_token_secret)
api = tweepy.API(auth)
retwts = api.retweets(id=665594047684300800, count=20)
for status in retwts
api.create_friendship(id=retwts)
I'm very new to programming sorry if that code was dumb but that was the best i could do by reading the docs, and it obviously isn't working. Can someone tell me where is the problem?
This is the error:-
File "tweepybot.py", line
for status in retwts
^
SyntaxError: invalid syntax
Thank you in advance.

For the syntax, make sure to end for statements with a colon, then indent the next lines. Also, within the indented part, you'll need to use the loop variable (status), rather than the list (retwts) you're looping over. Finally, the id argument must be given the id of the user:
for status in retwts:
    api.create_friendship(id=status.user.id)

If you have more problems, Stack Overflow is a good place for questions.

Dude, thank you so much, since i self taught myself python, this small thing was bugging me to death i stayed up till 5 AM, this helped me a LOT! Thanks again!

Also read the entire official Python tutorial if you haven't already.

This isn't a tweepy issue - please mark this closed.


Scheduler resiliency

This makes the scheduler methods gather, scatter, and schedule resilient to worker failure.

Sorry to have been ignoring this lately

Closing this.  The existing dask.distributed scheduler is informally deprecated.  Not planning to spend more time working on it.  Hope to have something easier to work with in 3-6 months time.


Remove Bidders with Video and Native Demand entries for bidders that …

…do not support Prebid 1.X

The table has always been like that, this update just removes the bidders that don't support 1.x. I created a new JIRA ticket to encompass further updates.


Router tree generation different between OSs (Ubuntu/macOS)

Version
v2.11.0
Reproduction link
https://codesandbox.io/s/zealous-ardinghelli-2610p
Steps to reproduce
I have this file structure:

When building a production bundle localy on macOS 10.15.3, the generated router.js is fine and as expected
{
      path: "",
      component: _07db9c7c,
      name: "index"
    }, {
      path: "communication",
      component: _0f75dec6,
      name: "index-communication"
    }, {
      path: "content",
      component: _17fb9be9,
      name: "index-content"
    }, {
      path: "statistics",
      component: _1256e1e3,
      children: [{
        path: "",
        component: _60459bb4,
        name: "index-statistics"
      }, {
        path: "analytics",
        component: _0591ecda,
        name: "index-statistics-analytics"
      }]
    }, {
      path: "users",
      component: _417c0bf8,
      name: "index-users"
    }

This is the content of statistics.vue file, with a tab component to swich between childs
<template>

    <Container class="h-center p-5 p-w-10 nowrap nogrow w-between">

{{ selectedTab ? selectedTab.label : '' }}

      <Tabs :tabs="tabs" :selected.sync="selectedTab" />
    </Container>

      <transition name="page" mode="out-in">
        <NuxtChild />
      </transition>

</template>

What is expected ?
When I do a nuxt build on Ubuntu, I want the router tree to be the same as it is locally, with children correctly generated
What is actually happening?
This is the the generated ouput of router.js on an Ubuntu machine after a nuxt build
{
      path: "",
      component: _5feafe74,
      name: "index"
    }, {
      path: "Communication",
      component: _b3c7cd50,
      name: "index-communication"
    }, {
      path: "Content",
      component: _4c3dabfb,
      name: "index-Content"
    }, {
      path: "statistics",
      component: _035c3b98,
      name: "index-statistics"
    }, {
      path: "Statistics",
      component: _66230091,
      name: "index-Statistics"
    }, {
      path: "Users",
      component: _1cc6f88a,
      name: "index-Users"
    }, {
      path: "statistics/analytics",
      component: _ea183430,
      name: "index-statistics-analytics"
    }

The children routes are removed are replaced by single routes, breaking the app ,all links and structure

This bug report is available on Nuxt community (#c10305)

The Ubuntu output has capitalized path names, which is quite strange and most likely also the cause of the problem. Netlify uses Ubuntu 16.04 for builds and they work fine, so it must be something in your project or on your server.

@matteo-rigon It was a really frustrating bug, it somehow fixed itself when I changed the name of the parent route...


Fix gunicorn command

I tried to push the code in the cloud directory of the branch add-bootstrap with the command git subtree push --prefix cloud heroku add-bootstrap:main.
The log (viewed with heroku logs --tail):
2022-02-08T00:00:04.247835+00:00 heroku[web.1]: Starting process with command `gunicorn app:app --preload --log-level=debug --bind <IP_ADDRESS>:${PORT} wsgi`
2022-02-08T00:00:05.319691+00:00 app[web.1]: usage: gunicorn [OPTIONS] [APP_MODULE]
2022-02-08T00:00:05.319764+00:00 app[web.1]: gunicorn: error: unrecognized arguments: wsgi
2022-02-08T00:00:05.493880+00:00 heroku[web.1]: Process exited with status 2
2022-02-08T00:00:05.624258+00:00 heroku[web.1]: State changed from starting to crashed

The gunicorn command only requires an app module and options but clearly there are two non-option arguments: app:app and wsgi. I have no idea what the latter one means but the first one looks like the gunicorn examples I have seen, so I decided to remove the latter one from cloud/Procfile. After that, the command git subtree push --prefix cloud heroku add-bootstrap:main successfully started the webserver.

cherry picked onto heroku-felix


Ticketing form inputs are apparently not required

Someone submitted without lastname and email.

Not sure what happened (I have only put HTML side checks), but ticketing is closing tomorrow so I think we can close this issue.


fix: inefficiencies in processing Agora contexts

This changes the Agora context to only rely on a &str into the client request document to avoid unnecessary String allocations, and removes the need to copy the context for every indexer selected. Most of the changes are dedicated to indexer request rewrites, to avoid the need for Context<_, String>, while relying on graphql_parser to serialize everything below the top-level selection set fields.

See also: https://github.com/graphprotocol/agora/pull/64


Dictionary: try(key) method to return value or null when key doesn't exist.

Actual behavior:
Raise a error 'Invalid get index 'key_name'
New behavior:

Keep raising error but dictionaries will have the method 'try(key_name)' which returns value if key exists or returns null if the key doesn't exist

This will clean the code. The multiples lines with if my_dictionary.has('key_name'): my_variable = my_dictionary.key_name will no longer be necessary. It increases the developer happiness.

I agree with this one but I think it should be more similar to Python:
dict.get(key[, default]): which returns None or default if the key doesn't exist (it doesn't raise any error).
Docs: https://docs.python.org/3/library/stdtypes.html#dict.get

Does anyone mind if I attempt this one?  It's not currently assigned.  Are there any recommendations for testing builds without having to rebuild the entire project after each change, such as a way to run it in debug mode?

@ordidug Thanks.  I found the answer.
For anyone with a similar question in the future,  please see the following from the docs:  http://docs.godotengine.org/en/latest/development/cpp/custom_modules_in_cpp.html#improving-the-build-system-for-development
"However the trade-of is every single change means a full recompilation of the game. Even if SCons is able to detect and recompile only the file that have changed, finding such files and eventually linking the final binary is a long and costly part.
The solution to avoid such a cost is to build our own module as a shared library that will be dynamically loaded when starting our game’s binary."

Fixed by #20627.


Deploy to production

@backspace I made another commit to the last branch with some small additional fixes, but did something weird where it's trying to merge with main or something. :/

@backspace I made another commit to the last branch with some small additional fixes, but did something weird where it's trying to merge with main or something. :/

hmm, so does what’s in main represent what you want to be deployed? I’m not sure what happened… #53 was merged into main and then deployed but #56 is a continuation of #53 so there are conflicts? It’s sometimes preferable to open a new branch/PR after it’s been merged once vs adding to it, but I think this can be recovered from by deleting the production branch locally and doing a “force push” to GitHub, something like this:
git checkout main
git pull main
git branch -D production
git checkout -b production
git push -uf origin production

(Assuming the GitHub remote is called origin)
Usually force pushes are frowned upon though… merging might also work because I think you can choose the versions of those three files that come from #56 without having to actually merge manually 🤔

hmm, so does what’s in main represent what you want to be deployed? I’m not sure what happened… #53 was merged into main and then deployed but #56 is a continuation of #53 so there are conflicts? It’s sometimes preferable to open a new branch/PR after it’s been merged once vs adding to it, but I think this can be recovered from by deleting the production branch locally and doing a “force push” to GitHub, something like this:
git checkout main
git pull main
git branch -D production
git checkout -b production
git push -uf origin production

(Assuming the GitHub remote is called origin)
Usually force pushes are frowned upon though… merging might also work because I think you can choose the versions of those three files that come from #56 without having to actually merge manually 🤔


minLength option doesn't default to 1

In the documentation the property minLength states that its default value is 1, but in the latest version it seems more likely that this value is 5.
https://github.com/twitter/typeahead.js/blob/master/doc/jquery_typeahead.md#options
Should the documentation be updated o or change the default value back to 1?

For me it seems that in v0.11.1 the minLength is defaulted to 1, as stated by the documentation.


[SDPL-169] Adds border radius to option group

JIRA issue: https://digital-engagement.atlassian.net/browse/SDPA-
Changed

Add border-radius to first and last item in option group at Desktop BP
Fix clear button text color
Fix export for list-group item

Screenshots

Hey @tim-yao, have fixed those vars. I've also fixed up the label reference, the labels weren't joined up with the inputs correctly. But it looks like vue-form-generator isnt  doing it right with its own core default radio group component https://github.com/vue-generators/vue-form-generator/blob/master/src/fields/core/fieldRadios.vue It needs to have an ID that matches the label for attribute. If I get around to it i'll submit a PR to VFG. FYI VFG hasn't had any activity for a loooong time.
Current -

Should be

Notice the id on field-wrap and the numbered label input pairs

Hey @tim-yao, have fixed those vars. I've also fixed up the label reference, the labels weren't joined up with the inputs correctly. But it looks like vue-form-generator isnt doing it right with its own core default radio group component https://github.com/vue-generators/vue-form-generator/blob/master/src/fields/core/fieldRadios.vue It needs to have an ID that matches the label for attribute. If I get around to it i'll submit a PR to VFG. FYI VFG hasn't had any activity for a loooong time.

Current -

Should be

Notice the id on field-wrap and the numbered label input pairs

Thanks @dylankelly . We can try submit a PR to VFG and see if they will take it. If they stop to maintain it, that's bad news.
For radios name, we need to keep same within one group, otherwise BE(Drupal webform) won't able to recognise the dynamic form data name. Same as the new OptionButton I think, even it's not post back to Drupal webform, it may will be an issue in handle the submission.
<legend>Output format</legend>

    <input type="radio" name="format" id="txt" value="txt" checked>
    Text file

    <input type="radio" name="format" id="csv" value="csv">
    CSV file

  […]
</fieldset>```


Error in swift 3.0

I have an error in file: DropDown.swift in line 807 => self.isHidden = true

DropDown has since been updated to swift 3.0. Please download the last version of DropDown.

I update the pod but i have the same error:
error in file: DropDown.swift in line 807 => self.isHidden = true

What is the compilation error message, please?

Yeah same error here..!!

Same error here. Its unusable.

same error here...

I have the same issue.

I have the sammmmeeeeee issue.

I figured out that I had stored weak link to it and it deallocated before full initialization. Make sure that you have strong ref to it.

could you post an example here?

final class Some: UIViewController {
    var dropDown: DropDown?
    func setupDropDown() {
        if let typeContainer = container {
            let dropDown = DropDown()
            dropDown.anchorView = typeContainer
            dropDown.dataSource = ["", "", ""]
            ................................................
            dropDown.direction = .bottom
            dropDown.shadowColor = UIColor.clear
            dropDown.backgroundColor = UIColor.white
            dropDown.selectionBackgroundColor = UIColor.clear
            dropDown.cellNib = UINib(nibName: "MenuCell", bundle: Bundle.main)
            dropDown.bottomOffset = CGPoint(x: 0, y: typeContainer.bounds.height)
            self.dropDown = dropDown
        }
    }

    override func viewDidLoad() {
        super.viewDidLoad()
        ........................
        setupDropDown()
    }

    @IBAction func onTypeChange() {
        if let typeButton = typeButton {
            typeButton.isSelected = !typeButton.isSelected
            if typeButton.isSelected {
                self.dropDown?.show()
            } else {
                self.dropDown?.hide()
            }
        }
    }
}

i have same error....


Typed text starts on next line down from cursor click

Challenge Create a Bootstrap Headline has an issue.
User Agent is: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:53.0) Gecko/20100101 Firefox/53.0.
Please describe how to reproduce this issue, and include links to screenshots if possible.
This issue occurs on challenges where the code element occupies most of the height of the screen.
On loading a new challenge page, click on the coding window.  The vertical line indicating the current position of text writer appears.
Use any keyboard input.
Typing will jump to the line immediately below the original click and apparent position of the text writer (vertical line)

Duplicate of #7847


Clean up focus background implementation

Description
Clean up some of the focus background implementation details

Make ouiFocusBackground mixin actually use $color argument
Make $ouiFocusTransparency an actual percentage instead of a float [0, 1]
Refactor any usage of $ouiFocusTransparency to respect it as a percentage instead of a float [0, 1]

Before

After

Check List

[ ] New functionality includes testing.
[ ] New functionality has been documented.
[x] All tests pass

[x] yarn lint
[x] yarn test-unit

[ ] Update CHANGELOG.md
[x] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

I changed the shade amount to 65%. It seems to be a little better but I cant check with next theme because some of the colors are detected as dark when they should be light and vice versa

Does this address https://github.com/opensearch-project/oui/issues/940 ? If so, LGTM!!!

Yep, also changes contrast :)


Not clear enough

My partner didn't understand the previous sentence.
From "Directory account:  Your customer can create a new user from your organization in their directory and assign any RBAC role."
To "Directory account:  Your customer can create a user account for you in their own directory and assign any RBAC role."

@jeanrobertjs : Thanks for your contribution! The author, @dhirajgandhi, has been notified to review your proposed change.


memory_backend.py update_manifest() is expecting all new_obj STIX objects to have a "modified" element.  Not all STIX 2.0 objects do.

It appears that update_manifest() requires all STIX objects to have a "modified" element, that it uses  to update the STIX object's version history.  If there is not a modified element present, it throws an error.
However, not all STIX 2.0 objects have a "modified" element.  For example, a Marking-Definition STIX 2.0 object cannot have a "modified" element.  If your STIX bundle contains a Marking-Definition object, update_manifest() will throw an error.

Thanks, @cricard . I feel like we ran into a similar issue lately, but I can't find it. I agree this needs to be fixed.

I asked a question on the TAXII slack channel on ideas on how to implement this.  Of course, no one answered the original question.

Try the public list vs. non-public Slack Channels?

Because "versions" is an optional property, the fix will be just to not populate that property when the "modified" property is not present.

After discussion on slack, it was decided to include the "created" timestamp for the one and only version of a Marking-Definition manifest entry


JSON Theme

Convert Theme structure from Typescript to JSON format.
This has 2 main advantages:

Validate theme values and structure at build time.
Possibility to extract Theme (JSON files), outside the codebase (such as in a DB, remote server).

How it works?
We use JSON Schema to validate Theme JSON format.
However, since writing a JSON Schema can be lengthy and has a bit of a learning curve, we use Typescript to generate the JSON Schema. To do so, we use typescript-json-schema.
We also use ajv-cli to validate that JSON Theme files follow the defined shema. These tools are included in the build script to make sure everything will work correctly.
One possible improvement could be to be able to extend themes, a bit like before. For example, the base light theme could contain all theme keys and values, while dark theme could only specify the values it wants to override from the light theme.

@hansgx Opened as Draft to start discussion on this :wink:


remove redundant catch around GetConnectionProperties

paging @Haplois

paging @Haplois

Yes, removing that should be OK; I couldn't find a good reason for that catch.


[Isthmus] Specify failure behavior on cast

The field Cast::failure_behavior has no listed default and yet Isthmus does not specify a value which forces consumers to guess.

Hello - I was just looking at this as it's occurred in some testing I was doing (and had to manually modify the substrait plans to workaround it)
Expressions is created here
  public static Function<TypeConverter, SimpleCallConverter> CAST =
      typeConverter ->
          (call, visitor) -> {
            if (call.getKind() != SqlKind.CAST) {
              return null;
            }

            return ExpressionCreator.cast(
                typeConverter.toSubstrait(call.getType()),
                visitor.apply(call.getOperands().get(0)));

          };

Does it make sense to add a field/method on the TypeConverter to specify the error condition?
Extra field needs to be added to the ExpressionCreate.cast   - question is where to get this from...

My guess (which could be very wrong given how little I know about isthmus) is that it call.getKind() == SqlKind.CAST implies FAILURE_BEHAVIOR_THROW_EXCEPTION and call.getKind() == SqlKind.SAFE_CAST implies FAILURE_BEHAVIOR_RETURN_NULL.  Although, from the snippet you provided, it looks like SAFE_CAST isn't being handled here anyways.

Admit I've not heard of a SAFE_CAST  might be the version of calcite-sql I've got.
Would make sense to me to have the default be throw an exception.

TIL about SqlKind.SAFE_CAST vs SqlKind.CAST. I think Weston has it correct in terms of the mappings.
@mbwhite if you were to make a PR for this I'd be happy to review it.


Alias ignored when flattening

Given an enum with a name and an alias, such as below.
#[derive(Debug, Serialize, Deserialize, Clone)]
pub enum PSCNatureOfControls {
    #[serde(
        rename = "{http://xmlgw.companieshouse.gov.uk}NatureOfControls",
        alias = "{http://www.govtalk.gov.uk/CM/envelope}NatureOfControls"
    )]
    NatureOfControls(NatureOfControls),
    #[serde(
        rename = "{http://xmlgw.companieshouse.gov.uk}LLPNatureOfControls",
        alias = "{http://www.govtalk.gov.uk/CM/envelope}LLPNatureOfControls"
    )]
    LLPNatureOfControls(LLPNatureOfControls),
}

Then put in a struct as such, and flattened.
#[derive(Debug, Deserialize)]
pub struct CompanyDataPSCNotification {
    #[serde(flatten)]
    pub nature_of_control: PSCNatureOfControls,
    ...
}

And when an enum value with the alias name is presented to the deserializer, an error such as "no variant of enum PSCNatureOfControls found in flattened data" is given. When the enum variant is changed from alias to rename, it works as expected.

Duplicate of #1504.

This was fixed (partially) by #2387, the in-place part is fixed in #2443


Profiles: duplicate (Michaela Ziegler, ahdis)

ch.fhir.ig.ch-etoc#1.1.0 /profiles.html
The profile 'CH eTOC Care Plan' is listed twice
Michaela Ziegler, ahdis

Solved

Solved

Checked and issue closed.


 does  popups are redundant

The popup is already attached to  so it just makes the messages unnecessarily long. Just say "Did "

If there's too many people, or if ever pop-ups get logged to some chat category having the name can be useful.

If it's logged it should definitely include the name, but right now it makes it harder to read, especially if they're moving and they have a long name.

It could work well in instances like pickpocketing or stripping. Or you could be in a group and it helps confirm what you're doing to who.


plug installation snippet seems wrong?

Hello, nice theme, I think the snippet for installing with plug is wrong, the .kak  seems to be lacking. ;)

You're absolutely right. I'll fix this immediately. Thanks for the help :)

@eko234 Fixed!


Add new locales using the CLDR dataset

Mojito's locale list is now composed of all available locales in CLDR + all the languages that have an entry with a display name in CLDR.
The main different compared to previous data set is that now, languages only are also available. Before that, all locales had a region (beside "en" that was the root locale).
For example, you now have "fr" and "fr-FR".
This also update the frontend to be able to render all locale names (including with scripts).

@patcon this cover most of your list, only "yupik" is missing which seems to be a group of locale.

Looks amazing, Jean! But given the number of operations, might be worth consolidating into BULK INSERT? On my mind because hosted apps like heroku often only have so many seconds (90 sec for heroku) to bind a port, and startup is already 60-80 second on first go :)


[$125] Action menu - Download option is missing from action menu

If you haven’t already, check out our contributing guidelines for onboarding and email<EMAIL_ADDRESS>to request to join our Slack channel!

Version Number: v9.0.18-7
Reproducible in staging?: Y
Reproducible in production?: Y
If this was caught during regression testing, add the test name, ID and link from TestRail: https://expensify.testrail.io/index.php?/tests/view/4836901
Logs: https://stackoverflow.com/c/expensify/questions/4856
Issue reported by: Applause-Internal team
Action Performed:

Log out of New Expensify
Navigate to a public room (https://staging.new.expensify.com/r/2091104345528462
) as an anonymous user
Right click on any attachment to access the action menu

Expected Result:
There should be an option to download the attachment.
Actual Result:
Option to download attachment is missing from action menu.
Workaround:
Unknown
Platforms:

Which of our officially supported platforms is this issue occurring on?

[ ] Android: Native
[x] Android: mWeb Chrome
[ ] iOS: Native
[x] iOS: mWeb Safari
[x] MacOS: Chrome / Safari
[ ] MacOS: Desktop

Screenshots/Videos
https://github.com/user-attachments/assets/120693aa-c566-44a2-bd92-4216575c71fa

View all open jobs on GitHub
Upwork Automation - Do Not Edit

Upwork Job URL: https://www.upwork.com/jobs/~018493448a5422f7d5
Upwork Job ID:<PHONE_NUMBER>486929052
Last Price Increase: 2024-08-09
Automatic offers:

brunovjk | Reviewer | 103540677
Krishna2323 | Contributor | 103540678

Issue OwnerCurrent Issue Owner: @yuwenmemon

Man that room takes a minute to load

Making this just $125 because this doesn't actually prevent the behaviour though (ie, you can still download from the big attachment modal) and public rooms are not well used yet,s o not a huge company priority.

I tested the proposed solution, and removing common.download from the restricted actions list does fix the issue. However, I want to confirm the expected behavior. Should we remove download from all or implement a dynamic approach, as suggested in the alternative solution? Anyway I’m leaning towards @Krishna2323’s proposal but wanted to check in.
🎀👀🎀 C+ reviewed

Ah, apologies, I was OOO - thanks for the ping! Assigned 🙂

Hey @yuwenmemon :D have you had a chance to look at the alternative solution here? I believe the main solution will work fine, but I wanted to hear your thoughts. Thanks.

I do not think we should prevent downloads for anonymous users personally. If I want to download a picture of this cat, why not?

I agree though that we have some disparity between the menu options for an authenticated user and a restricted user.

Shouldn't we just show the download option to all users?

Agreed with @twisterdotcom

[!NOTE]
The production deploy automation failed: This should be on [HOLD for Payment 2024-09-06] according to https://github.com/Expensify/App/issues/48035 prod deploy checklist, confirmed in https://expensify.slack.com/archives/C01GTK53T8Q/p1725051480370659?thread_ts=1725050431.962859&cid=C01GTK53T8Q.

cc: @twisterdotcom

[x]  [@brunovjk] The PR that introduced the bug has been identified. Link to the PR: The code that introduced the bug was present since the creation of the src/pages/home/report/ContextMenu/ContextMenuActions.tsx file. No specific PR was found that later modified this line. Link to file in creation commit

[x]  [@brunovjk] The offending PR has been commented on, pointing out the bug it caused and why, so the author and reviewers can learn from the mistake. Link to comment: Since the bug was not introduced by a specific PR but has been present since the file's creation, there is no PR to comment on.

[x]  [@brunovjk] A discussion in #expensify-bugs has been started about whether any other steps should be taken (e.g., updating the PR review checklist) in order to catch this type of bug sooner. Link to discussion: No discussion is necessary on Slack as this issue is isolated and was caused by a historical oversight rather than a review process flaw. The regression test should suffice to prevent recurrence.

[x]  [@brunovjk] Determine if we should create a regression test for this bug. Yes

[x]  [@brunovjk] If we decide to create a regression test for the bug, please propose the regression test steps to ensure the same bug will not reach production again.

Regression Test Proposal

Log out of the app.
Navigate to a public room as an anonymous user (https://staging.new.expensify.com/r/2091104345528462).
Right-click on any attachment to access the action menu.
Verify that the download option is visible in the action menu.
Perform the same steps while offline and confirm the download option is still visible.

Do we agree 👍 or 👎

Payment Summary:

@brunovjk paid $250 via Upwork here (C+)
@Krishna2323 paid $250 via Upwork here (Contributor)


Use Python 3.12 as reference environment

Blocked by and incorporates #10891
Follow-up: #10893

thx


Explore API standarization

Profiler, Keycollision, Outliers. JDBC does not have a standard way to be used.

@argenisleon Can you elaborate a bit more about the work required here? Maybe I can help  :)

At the moment Novembre 2019 the API seems very stable. We think we can improve how users access the Key Collision , Distance Cluster and encoding functions.
For the profiler, JDBC and Outliers we have:

Using Optimus object

Profiler
op.profiler.run()

JDBC
db = op.connect(
    driver="mysql",
    host="<IP_ADDRESS>",
    database= "optimus",
    user= "test",
    password = "test")

Using Dataframe object:

Outliers
df.outliers.tukey("age").select()

What can be improved
Key Collision and distance cluster and encoding
The string_to_index and bucketizer functions are already exposed. I think we can improve how to access for example fingerprint_cluster
At the moment the user needs:
from optimus.ml import keycollision as keyCol
df_kc = keyCol.fingerprint_cluster(df, 'STATE')

and could be written
df.ml.fingerprint_cluster('STATE')

@eschizoid What do you think?

@argenisleon It makes sense, but let me familiarize myself with these API's i can prepare a more elaborate answer :)
I would definitely be happy taking this ticket though...

All this has been refactored in 3.0.


SDP-927 - [SEP24] add configuration for multi-tenant registration flow

What
This configuration was used to test wallet registration multi-tenancy.

SEP10_HOME_DOMAINS: "redcorp.stellar.local:8000,bluecorp.stellar.local:8000" defines the two tenants in the allowable home domains.
SEP10_HOME_DOMAIN: "" is needed as a workaround to bypass anchor's default values.
SEP10_WEB_AUTH_DOMAIN: "localhost:8080" defines the proper web auth domain.

Tested that using the multi-tenant SDP as a home domain instead of the anchor works properly. The multi-tenant home domain is properly passed in the JWT token could be used to identify the tenant and register the wallet appropriately.

Notes

The example tenants used in SEP10_HOME_DOMAINS are actually coming from the dev/main.sh startup script that creates two tenants redcorp and bluecorp.

Known limitations
[TODO or N/A]
Checklist
PR Structure

[ ] This PR has a reasonably narrow scope (if not, break it down into smaller PRs).
[ ] This PR title and description are clear enough for anyone to review it.
[ ] This PR does not mix refactoring changes with feature changes (split into two PRs otherwise).

Thoroughness

[ ] This PR adds tests for the new functionality or fixes.
[ ] This PR contains the link to the Jira ticket it addresses.

Configs and Secrets

[ ] No new CONFIG variables are required -OR- the new required ones were added to the helmchart's values.yaml file.
[ ] No new CONFIG variables are required -OR- the new required ones were added to the deployments (pr-preview, dev, demo, prd).
[ ] No new SECRETS variables are required -OR- the new required ones were mentioned in the helmchart's values.yaml file.
[ ] No new SECRETS variables are required -OR- the new required ones were added to the deployments (pr-preview secrets, dev secrets, demo secrets, prd secrets).

Release

[ ] This is not a breaking change.
[ ] This is ready for production.. If your PR is not ready for production, please consider opening additional complementary PRs using this one as the base. Only merge this into develop or main after it's ready for production!

Deployment

[ ] Does the deployment work after merging?

stellar-disbursement-platform-backend-preview is available here:SDP:      https://sdp-backend-pr148.previews.kube001.services.stellar-ops.com/healthAP:       https://sdp-ap-pr148.previews.kube001.services.stellar-ops.com/healthFrontend: https://sdp-backend-dashboard-pr148.previews.kube001.services.stellar-ops.com

stellar-disbursement-platform-backend-preview is available here:SDP:      https://sdp-backend-pr148.previews.kube001.services.stellar-ops.com/healthAP:       https://sdp-ap-pr148.previews.kube001.services.stellar-ops.com/healthFrontend: https://sdp-backend-dashboard-pr148.previews.kube001.services.stellar-ops.com

stellar-disbursement-platform-backend-preview is available here:SDP:      https://sdp-backend-pr148.previews.kube001.services.stellar-ops.com/healthAP:       https://sdp-ap-pr148.previews.kube001.services.stellar-ops.com/healthFrontend: https://sdp-backend-dashboard-pr148.previews.kube001.services.stellar-ops.com

stellar-disbursement-platform-backend-preview is available here:SDP:      https://sdp-backend-pr148.previews.kube001.services.stellar-ops.com/healthAP:       https://sdp-ap-pr148.previews.kube001.services.stellar-ops.com/healthFrontend: https://sdp-backend-dashboard-pr148.previews.kube001.services.stellar-ops.com


JavaScript heap out of memory

Hi,
I try to analyze my app, but unfortunately I still receive FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory after long time waiting for analising data. Traceevent file is 1,87 GB.
I generally run my application and run for about 4/5 minutes e2e tests.
Analysing data

<--- Last few GCs --->

[24700:00000212E1D051E0]  7831109 ms: Mark-sweep 1802.1 (1852.7) -> 1802.1 (1838.7) MB, 2021.3 / 0.0 ms  last resort GC in old space requested
[24700:00000212E1D051E0]  7832953 ms: Mark-sweep 1802.1 (1838.7) -> 1802.1 (1838.2) MB, 1843.0 / 0.0 ms  last resort GC in old space requested

<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 000001DED02A5879 <JSObject>
    1: _onToken [C:\Projekty\Learn\node-clinic\node_modules\jsonparse\jsonparse.js:~342] [pc=0000032ABCACE55F](this=00000067E07A0029 <Parser map = 00000185D9FA8E21>,token=1,value=000002E504BEDBD9 <String[1]: {>)
    2: onToken [C:\Projekty\Learn\node-clinic\node_modules\JSONStream\index.js:~124] [pc=0000032ABCAD0AE0](this=00000067E07A0029 <Parser map = 00000185D9FA8E21>,token=1,value=000002E50...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
 1: node_module_register
 2: v8::internal::FatalProcessOutOfMemory
 3: v8::internal::FatalProcessOutOfMemory
 4: v8::internal::Factory::NewUninitializedFixedArray
 5: v8::internal::WasmDebugInfo::SetupForTesting
 6: v8::internal::WasmDebugInfo::SetupForTesting
 7: v8::internal::WasmDebugInfo::SetupForTesting
 8: v8::internal::WasmDebugInfo::SetupForTesting
 9: v8::internal::SharedFunctionInfo::SetScript
10: v8::internal::JSReceiver::class_name
11: v8::internal::SourcePositionTableIterator::Advance
12: v8::internal::SourcePositionTableIterator::Advance
13: 0000032ABC8043C1

Could you please guide me how I can make this analyze to work.

Hi Lukas, in the short term, I'd suggest trying a shorter benchmark - maybe 30 seconds initially, then try a minute, if that works try 1 minute 30, etc. If you can report back with the longest and largest (duration and mb of trace event file) profile you're able to successfully generate, that'd be really helpful.
Usually with clinic, it's possible to get useful and meaningful information from profiles shorter than a minute. The intention is that for most cases, a user should be able to get usable clues on where to next focus their optimisation efforts from profiles as short as 10 seconds (though obviously there will be cases where this isn't appropriate).
More long term, supporting larger profiles is a current priority.
@mafintosh do you think porting an equivalent of https://github.com/nearform/node-clinic-bubbleprof/pull/241 to Doctor would be a quick way to give something more useful in such cases?

@mcollina @mafintosh instead of stop inspecting the trace events file, we could also go ahead and just stop collecting any further trace events after reaching a specific number of events. That way the actual inspected events would likely be higher than just stop visualizing them when the heap reaches a specific heights. I would go so far and let doctor close the running application on it's own and to print a warning why that has happened. What do you think?


What's the prefered way to add a gap between items?

My only tought is to add an additional div arround the items including a bottom-padding. Is there a simpler/prefered solution?

follow

bump

bump

bump


Example of using Dhall to generate terraform file to manage a Github Organisation

Hi,
No treally an issue, but I have created an example of using Dhall to generate terraform file to manage a Github Organisation.
https://github.com/etaty/example-dhall-terraform-github
The Dhall community is small, so I thought sharing it here would be a good idea.
Also thanks for creating this repo, it helped me get started to use Dhall with Terraform 👍

Excellent! I'm glad you found it helpful 😄
One of the challenges I ran into was that earlier versions of the Dhall compiler were extremely slow (~1 minute to evaluate the interface I've described for datadog, for example).
I still believe that the right thing to do is to describe the API with as much rigor as possible, (very few primitive fields, choosing to express Text as an enumeration with smart constructors) with the hope that the compiler will get even faster, and that tooling will get better as well.
I'm still working on DataDog support in the datadog-provider branch, and I'm generally hoping to carve out support for at least the subset of the Terraform providers that people in the Dhall community actually care to use.
I will say, there's one major downside of the approach I'm taking, at least currently, which is the verbosity of abstractions that are necessary to provide explicitly, as the definitions are done without explicitly writing a dhall-terraform tool that integrates with dhall-haskell. This is done because I suspect it would be much more fruitful to implement (or cross-compile) a Dhall compiler in Go, then implement .tf.dhall directly in Terraform itself. Once that work is done, I expect most of the abstractions I am attempting to build here will map to that new platform without too much difficulty. Unfortunately, I have no interest in Go, nor do I currently feel as though I would be able to contribute to the Terraform codebase in a way that would conform to idiomatic Go, thus doing more harm than good.
If you'd like to contribute to the set of definitions in this repository, I'd gladly welcome PRs! Especially right now, I think consolidation in distinct implementation strategies will be helpful as the community grows.


Can't import this extension (json missing)

If I try to import this xsix file VS Code says "extension/package.json not found inside zip."

This is a Visual Studio IDE extension - not a VSCode extension.


najdorf-1.3 unable to access during Get Started Guide

Heya - Apologies if this is the wrong place to ask this, I'm more than happy to re-ask in the appropriate place with guidance.
I'm following the Get Started guide attempting to use the Docker deployment path for both local and Ubuntu 22.04 deployment of the pulled najdorf-1.3 and najdorf-latest images and in both circumstances end up with the bridge port 8443 unable to resolve anything on the container. I'm hoping y'all can help me because I would love to audit this tool as an option for my companies needs.
I'm attaching the docker desktop log from a fresh pull of the 1.3 container, and otherwise would love any guidance you can offer.
1.3-startup.log

We're missing some httpd config files that are required for the all-in-one container.

Is there another version of the tool I can try out that is know to be working? Had bad luke with all the najdorf versions and morphy


tls issues after updating to debian buster

Hi
After updating to Debian buster bimmer_connected is unable to establish a ssl connection to "b2vapi.bmwgroup.com".
The error message i get here is:
requests.exceptions.SSLError: HTTPSConnectionPool(host='b2vapi.bmwgroup.com', port=443): Max retries exceeded wit                                               h url: /gcdm/oauth/token (Caused by SSLError(SSLError(1, '[SSL: UNSUPPORTED_PROTOCOL] unsupported protocol (_ssl.                                               c:1056)')))
As the error "SSL: UNSUPPORTED_PROTOCOL" seems weird to me i investigated further.
It seems that b2vapi.bmwgroup.com only supports TLSv1.0
-> https://www.ssllabs.com/ssltest/analyze.html?d=b2vapi.bmwgroup.com
Also the openssl version packaged in Debian Buster has TLSv1.0 Support disabled by default.
-> https://lists.debian.org/debian-devel-announce/2017/08/msg00004.html

Is your HA instance running on Debian Buster and the component can't get the data from the BMW Connected Drive portal anymore?

It started working for me after allowing TLSv1.0 in my container:
sed -i "s|TLSv1.2|TLSv1.0|g" /etc/ssl/openssl.cnf
However, that is not a great fix for this issue.

Not a great fix indeed, but seems to be the only solution as long as the BMW CD portal only supports TLS v1.0.


confirm the ports for board and kuberentes

Request context

A clear and concise description of what the bug is.
confirm the firewall ports
A clear and concise description of what you expected to happen.
we can open the firewall and just open the ports board uses.
Additional context
Version of board (last git commit number), OS version, docker/docker-compose engine version, kubernetes version.

OS:        [e.g. CentOS 7.8]
Board:     [e.g. 7.6-22-g797d93f (version-commit times-commit number)]
Docker:    [e.g. 19.03.0]
Docker-compose: [e.g. 1.17.1]
Kubernetes [e.g. v1.18.3]

Config files of board, you can get them by packaging "board.cfg" and files in the same directory, including subdirectory.
Log files, you can get them by package the /var/log/board/.
If applicable, add screenshots to help explain your problem.
Other necessary information.

22/tcp: SSH (master and worker nodes)
179/tcp: Calico BGP Port (master and worker nodes)
2376/tcp: Node driver Docker daemon TLS port (master and worker nodes)
2379/tcp: Kubernetes etcd server client API (on master nodes in multi-master deployments)
2380/tcp: Kubernetes etcd server client API (on master nodes in multi-master deployments)
4789/udp: Flannel VXLAN overlay networking on Windows cluster (master and worker nodes)
6443/tcp: Kubernetes API server (master nodes)
6783/tcp: Weave Port
6783-6784/udp: Weave UDP Ports
8472/udp: Flannel overlay network, VxLAN backend (master and worker nodes)
9099/tcp: Canal/Flannel livenessProbe/readinessProbe (master and worker nodes)
9100/tcp: Default port required by Monitoring to scrape metrics from Linux node-exporters (master and worker nodes)
9796/tcp: Default port required by Monitoring to scrape metrics from Windows node-exporters (master and worker nodes)
10250/tcp: Kubernetes kubelet API server (master and worker nodes)
10251/tcp: Kubernetes kube-scheduler (on master nodes in multi-master deployments)
10252/tcp: Kubernetes kube-controller-manager (on master nodes in multi-master deployments)
10254/tcp: Ingress controller livenessProbe/readinessProbe (master and worker nodes)
10255/tcp: Kubernetes kubelet API server for read-only access with no authentication (master and worker nodes)
30000-32767/tcp udp: Nodeports (master and worker nodes)

board ports:
80/tcp udp: web ui
443/tcp udp: web ui
8080/tcp udp: k8s proxy
1541/tcp: log
8088/tcp udp: apiserver
9200/tcp: Elasticsearch
9090/tcp: prometheus.

registry节点：
5000/tcp: registry
22/tcp: ssh
jenkins节点：
22/tcp: ssh

grafana： 3000/tcp udp
kibana： 5601/tcp  udp
gogits: 10080/tcp  udp  or gitlab: 10088/tcp  udp


arch: arm: cortex_m: fault: Improve handling of fault escalation on armv6-m

The current implementation causes a lockup of the core when the exception originates from an invalid/unreachable pc. This fix first verifies on armv6-m and armv8-m.base that pc was in an expected runnable region, namely:

.text
.ramfunc
.itcm

Looks good to me, other than the requested change from Carlo. Thanks for putting this together.

@carlocaione Are you OK with merging this one as a first step, and sorting out a multi-arch or at least generic Arm version separately?
@ithinuel Want to make a new PR with this as a generic helper, not just Cortex M?

@carlocaione Are you OK with merging this one as a first step, and sorting out a multi-arch or at least generic Arm version separately?
@ithinuel Want to make a new PR with this as a generic helper, not just Cortex M?

totally fine to me


tests: workqueue tests overhaul adding new tests

During inspection of the workqueue tests, I find out testing gaps.
Decided to add new test cases that can improve Zephyr OS testing
quality.
Added new test cases:

test_work_item_supplied_with_func
In docs described that work item supplied with a handler function,
prove that it works.
test_process_work_items_fifo
Test that system process work items in first-in, first-out manner.
test_sched_delayed_work_item
Verify that delayed work item processed after specific period of time
stated by user.
test_workqueue_max_number
Test the limit of number of workqueues created
test_cancel_processed_work_item Created test to increase branch
coverage.

Modified existing test cases:

test_work_submit_handler updated Doxygen tag, added more detailed
description"

Signed-off-by: Maksim Masalski<EMAIL_ADDRESS>

@nashif @andrewboie I'm looking for code review for that PR. Andy already gave comments.


Need to update group.id to "io.openscore"

After the change,
we need to upload the artifact to maven central @hararzafrir  , and then update the website @orius123

@hararzafrir can you upload the last artifact(of score-language) to maven central?

So i understand from @hararzafrir  it is now in Mvaen Central, we just need to verify iy works, and update the website, so i'm moving this to @orius123 ...

you changed only group id, not packages, right?

right


NestStack and AutoStartup of the CB and Glue Jobs

Issue #, if available:
Description of changes:
Added Nested stack for all the stacks
Made CodeBuild auto start the build
CDK was failing on second run of CDK because Kinesis was failing on because cdk was missing the delete policy on the stream.
ECR repo cleans up images
Made Glue jobs auto start and have a cdk.context.json flag called auto_start_glue_jobs with default to true.  In case dont want them running at start.
Removed the Readme manual steps for running the Codebuild and Glue
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

PR for nested stacks and autostartup of Glue and and CB


Fast track Identity Onboarding

Adding workflows for Fast track Identity Onboarding Requirement

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.sbharatiya seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.You have signed the CLA already but the status is still pending? Let us recheck it.


Used SNPedia to fill out / correct the data in the MPS JSON

I went through every entry in the MPS JSON file, and manually checked them against SNPedia, inputting the pathogenic genotypes and gene data.
I also added a couple SNPs associated with MPS per a particular Reddit comment.
I don't know if this is the sort of edit you want, but I did it for myself and now that I've done it, I figure I might as well see if upstream wants it.

I can squash the first three commits in this if you'd like.

This won't work well without #11, since it uses heterozygous genotypes and indels.
You can play around with both PRs at the same time by cloning https://github.com/MilesBHuff/snpeek.github.io/tree/main


Create new tab to show rectangles in a 2D space with arrows in between

Fixes #164

There are still some issues to be fixed with this:

Allow resizing of the canvas
Arrows change of anchor position too soon when dragging an element from a bottom position to the left.

Shall we extract issues for these?

Resizing of arrow calculation has been fixed, positioning the blocks is not. This is more complicated sadly. The arrows could be improved, but let's move that to a new issue.

Very good, I'll extract some issues


added ability to set job display name

fixes issue #23

See my comment in issue #23

Closing this request in favor of a different implementation based on JobSetting.


OBR updates from the official HL7 v2 to FHIR mapping

User Story:
As ResportStream, I want to follow the HL7 org's HL7 v2 to FHIR mapping as close as possible, so our FHIR data conforms to a community created spec.
Description/Use Case
We followed the published HL7 v2 to FHIR spec to generate our HL7 v2 to FHIR and FHIR to HL7 v2 conversions.  The original mappings are published via their HL7 v2 to FHIR mapping pages. HL7 has published updates to the mappings via published via Google Docs (these Google docs are now the official delivery of the mappings).  We want to update any mappings used by the UP now while we are not processing data as doing it later can impact data already processed by the pipeline.
Risks/Impacts/Considerations

The result of changing the mappings while we are processing data in UP is the same as updating the COVID 19 schemas.  Replaying a piece of data may fail as the mappings are different, or a mapping change while data is being processed may cause an issue in the data as well. A future plan to be able to update these mappings is needed, such as adding mappings to the HL7 v2 to FHIR conversion first for 60 days, then to the FHIR to HL7 v2 after.
Mappings labeled as under construction can be a risk, so bring up to the team for review before implementing them.

Dev Notes:

You need to carefully inspect our mapping docs vs the new mappings.
Comparing the mappings for a segment will take you down to inspecting the mappings for data types which are reusable in other segments, so coordinate with other developers inspecting other segments to limit duplicate work.
You will also need to verify extension names. E.g. some are at the original website linked from the resource that uses them.  We may be able to reduce the number of reportstream.cdc.gov or ibm.com extensions being used in favor of official extension names.
Some of the mappings are not possible either because of limitations in our conversion library or due to repetitions having multiple values for data that otherwise if supposed to go into just one FHIR property (e.g. Provenance to store multiple Observation values). See the comments in our mapping docs, but you may need to ask the team for feedback on those.
Skip any HL7 fields that we do not have data for, otherwise you have nothing to test with.
Do not change the source HL7 v2 test data for the integration tests as they are based on real data and we want to keep those as pristine as possible.

Acceptance Criteria

The OBR segments (this includes all repetitions) ReportStream mapping docs are updated with the latest HL7 v2 to FHIR mapping changes as appropriate. We need to make sure we are still functionally lossless, so some mappings may not be appropriate for this mission.
The Linux4Health OBR HL7 v2 to FHIR UP conversion mappings are updated with the latest HL7 v2 to FHIR mapping changes as appropriate.
The RS OBR FHIR to HL7 v2 UP conversion mappings are updated with the latest HL7 v2 to FHIR mapping changes as appropriate.
The Data Tests integration test data is updated to reflect the changes to the conversion mappings.

I'll be picking this one up from Jessica continuing where she left off.


Fix theme customizations caching

When using theme customizations on a concrete5 site I noticed an increased load time. After some investigation it seems that with theme customizations the CSS cache is actually not used.
The problem was introduced with this commit: https://github.com/concrete5/concrete5/commit/6bd27450c552c642b12ba68e9d3a206ce64527be#diff-7a93e2465b45c5883e9e24fa04035a3f
I'm not entirely sure which bug exactly this is supposed to fix, but it effectively disables all caching for pages with theme customizations. Before it would only use the "handler script" for previewing unapproved changes, after the "handler script" is used as soon as there are theme customizations.
I think the call $this->cp->canViewPageVersions() was removed by accident.

You raise some good points – I think this issue needs some more thought. I've opened a new issue for this.

@aembler Thanks for opening the new issue #3956 for further discussion about it - I agree rethinking it makes sense.
However the current behaviour in 5.7 has severe performance costs for our website as each page view causes the CSS to be recompiled. I think this should be fixed.
Any reason for not merging this PR?

Fair enough, this looks pretty low risk; I'm wary of merging it because as you've seen there's obviously been a number of rewrites to this and the logic is a little hairy, but I trust you've tested it. We'll leave the other issue open as well.

@aembler I did some more testing and trying to better understand the logic of PageView::getStyleSheet. I originally thought this worked before in 5.7 but I now think it never totally did, so I now consider this an improvement instead of a simple bug fix - sorry for change of minds.
I would recommend you either revert my PR or if you agree to fix this in 5.7 I can send another PR with the following code.
Basically it all works fine until someone presses the "Clear Cache" button in the dashboard (or copies the page as mentioned in 6bd27450c552c642b12ba68e9d3a206ce64527be): The caching of the customized CSS relies on the cached file to already exist - and that cache file only gets written after a change is made to the customization.
It's possible to fix this by writing that cached file if it isn't found, however that change on line 173 involves quite a bit of code and I understand if you consider that not worth doing for 5.7:
            } else if ($this->c->hasPageThemeCustomizations()) {
                $style = $this->c->getCustomStyleObject();
                if (is_object($style)) {
                    $scl = $style->getValueList();
                    if (is_object($scl)) {
                        $sheetObject->setValueList($scl);
                    }
                    $sheetObject->output();
                    return $sheetObject->getOutputRelativePath();
                }
            }

For the previously mentioned problem about the performance cost I guess I will have to resort to using a reverse proxy to forcibly cache the generated CSS.

@fabian Ah, thank goodness...my faulty brain would not recall the reason for those changes ;) That was indeed it – if the file was never written by the manual process in the first place it never would be.
I have reverted your change, since it had regressed behavior, but I would be interested in getting a working pull request that fixed the issue, and it looks like your code above would work fine. I don't think it's too much code to put in there to enable better caching of those files.

@aembler 😄 well there are many moving things involved there…
Thanks for reverting it!
I will see if I can send a PR - would you prefer it for the branch master or for the branch 5.7.x?

Let's put it against 5.7.x. I'm going to have to find a way to merge 5.7.x
into develop, so no need to open it against both.
On Tue, Jun 28, 2016 at 7:06 AM, Fabian Vogler<EMAIL_ADDRESS>wrote:

@aembler https://github.com/aembler 😄 well there are many moving
things involved there…
Thanks for reverting it!
I will see if I can send a PR - would you prefer it for the branch master
or for the branch 5.7.x?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/concrete5/concrete5/pull/3946#issuecomment-229059386,
or mute the thread
https://github.com/notifications/unsubscribe/AAgNwWBqBnf0RsGxCUWxyqDkyL33h30Bks5qQSpagaJpZM4I1DHV
.


Initial refToHTML implementation

First go (and no styling, I leave this to you, styling experts) :)
What we perhaps could do is to assemble an in-between CSL JSON, and use something like Citeproc.js to convert to HTML. That might be a bit more robust still.

Thanks for that! Mergin...
Will try to also use this in RefTarget component (see #29 )

Good job. Activated also for ref target editing dialog.


update evpn script

Require PR #53 to be merged first and code update to reflect inventory API change


RuntimeError: This attribute is not available during multi-objective optimization.

Description
NeptuneCallback crashes in multi-objective runs.
The following functions in optuna.visualization need a target to be specified for multi-objective runs:

plot_optimization_history
plot_parallel_coordinate
plot_param_importances
plot_slice
plot_contour
plot_edf

Accessing the following attributes of an optuna.study.Study object, during a multi-objective run, results in a RuntimeError being raised:

direction
best_value
best_params

Accessing the following attributes of an optuna.trial.FrozenTrial object, during a multi-objective run, results in a RuntimeError being raised:

value

To reproduce

or follow the code below.

Code snippet
import shutil
from argparse import Namespace

import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.utils.data as D
import torchvision as tv
import pytorch_lightning as pl
import torchmetrics as M
from pl_bolts.datamodules import CIFAR10DataModule

import thop
import optuna
import neptune.new as neptune
from neptune.new.integrations.pytorch_lightning import NeptuneLogger
from neptune.new.integrations.optuna import NeptuneCallback

SEED = 42
pl.seed_everything(SEED)

class TestModel(pl.LightningModule):
    def __init__(self, hparams):
        super().__init__()
        self.save_hyperparameters(hparams)

        _hidden = nn.Sequential(
            nn.Linear(512, 512),
            nn.ReLU()
        )
        self.mlp = nn.Sequential(
            nn.Flatten(),
            nn.Linear(self.hparams.input_size, 512),
            nn.ReLU(),
            *[_hidden for _ in range(self.hparams.num_hidden)],
            nn.Linear(512, self.hparams.num_classes),
            nn.ReLU()
        )

        _metrics  = M.MetricCollection({
            "accuracy": M.Accuracy(),
        })
        self.train_metrics = _metrics.clone(prefix="train/")
        self.val_metrics   = _metrics.clone(prefix="val/")
        self.test_metrics  = _metrics.clone(prefix="test/")

    def forward(self, x):
        return self.mlp(x)

    def loss_function(self, preds, targets):
        return F.cross_entropy(preds, targets)

    def configure_optimizers(self):
        return torch.optim.Adam(
            self.parameters(),
            lr=self.hparams.lr
        )

    def _shared_step(self, batch, batch_idx):
        data, targets = batch
        preds         = self(data)
        loss          = self.loss_function(preds, targets)
        return loss, preds, targets

    def training_step(self, batch, batch_idx):
        loss, preds, targets = self._shared_step(batch, batch_idx)
        metrics              = self.train_metrics(preds, targets)

        self.log("train/loss", loss)
        self.log_dict(metrics)
        return { "loss": loss }

    def validation_step(self, batch, batch_idx):
        loss, preds, targets = self._shared_step(batch, batch_idx)
        metrics              = self.val_metrics(preds, targets)

        self.log("val/loss", loss)
        self.log_dict(metrics)
        return { "loss": loss }

class Objective:
    def __init__(self, run, monitor):
        self.run     = run
        self.monitor = monitor

    def __call__(self, trial):
        tunable = Namespace(
            lr         = trial.suggest_loguniform("lr", 1e-5, 1e-1),
            num_hidden = trial.suggest_int("num_hidden", 0, 10, step=1),
            precision  = trial.suggest_categorical("precision", [16, 32]) if torch.cuda.is_available() else 32,
        )

        logger = NeptuneLogger(
            base_namespace = "my-run",
        )
        # neptune-pytorch-lightning#3 🙇🏽♂🙇🏽♂
        logger._run_instance = self.run

        dataset = CIFAR10DataModule(
            num_workers = 2,
            batch_size  = 32,
            seed        = SEED,
            shuffle     = True,
            pin_memory  = True,
        )
        hparams = Namespace(
            lr          = tunable.lr,
            input_size  = 32*32*3,
            num_hidden  = tunable.num_hidden,
            num_classes = 10,
        )
        model   = TestModel(hparams)
        trainer = pl.Trainer(
            gpus                    = -1 if torch.cuda.is_available() else 0,
            precision               = tunable.precision,
            max_epochs              = 4,
            num_sanity_val_steps    = 0,
            check_val_every_n_epoch = 1,
            deterministic           = True
        )

        trainer.fit(model, dataset)

        accuracy = trainer.callback_metrics[self.monitor].item()

        _inputs = (torch.randn(32, 3, 32, 32),)
        flops, num_parameters = thop.profile(model, inputs=_inputs, verbose=False)

        return flops, accuracy

run = neptune.init(
    api_token = "ANONYMOUS",
    project   = "common/new-pytorch-lightning-integration",
    name      = "sweep",
    mode      = "debug", # Comment to log online.
)
study = optuna.study.create_study(
    directions = ["minimize", "maximize"],
)
study.optimize(
    Objective(run, "val/accuracy"),
    n_trials  = 100,
    callbacks = [NeptuneCallback(run)]
)
run.stop()

Trace
---------------------------------------------------------------------------

RuntimeError                              Traceback (most recent call last)

<ipython-input-6-caa4acd21886> in <module>()
     11     Objective(run, "val/accuracy"),
     12     n_trials  = 100,
---> 13     callbacks = [NeptuneCallback(run)]
     14 )
     15 run.stop()

/usr/local/lib/python3.7/dist-packages/optuna/study.py in optimize(self, func, n_trials, timeout, n_jobs, catch, callbacks, gc_after_trial, show_progress_bar)
    408             callbacks=callbacks,
    409             gc_after_trial=gc_after_trial,
--> 410             show_progress_bar=show_progress_bar,
    411         )
    412

/usr/local/lib/python3.7/dist-packages/optuna/_optimize.py in _optimize(study, func, n_trials, timeout, n_jobs, catch, callbacks, gc_after_trial, show_progress_bar)
     73                 reseed_sampler_rng=False,
     74                 time_start=None,
---> 75                 progress_bar=progress_bar,
     76             )
     77         else:

/usr/local/lib/python3.7/dist-packages/optuna/_optimize.py in _optimize_sequential(study, func, n_trials, timeout, catch, callbacks, gc_after_trial, reseed_sampler_rng, time_start, progress_bar)
    174             frozen_trial = copy.deepcopy(study._storage.get_trial(trial._trial_id))
    175             for callback in callbacks:
--> 176                 callback(study, frozen_trial)
    177
    178         if progress_bar is not None:

/usr/local/lib/python3.7/dist-packages/neptune_optuna/impl/__init__.py in __call__(self, study, trial)
    146
    147     def __call__(self, study: optuna.Study, trial: optuna.trial.FrozenTrial):
--> 148         self._log_trial(trial)
    149         self._log_trial_distributions(trial)
    150         self._log_best_trials(study)

/usr/local/lib/python3.7/dist-packages/neptune_optuna/impl/__init__.py in _log_trial(self, trial)
    154
    155     def _log_trial(self, trial):
--> 156         _log_trials(self.run, [trial])
    157
    158     def _log_trial_distributions(self, trial):

/usr/local/lib/python3.7/dist-packages/neptune_optuna/impl/__init__.py in _log_trials(run, trials)
    443             handle[f'trials/{trial._trial_id}/state'] = repr(trial.state)
    444
--> 445         if trial.value:
    446             handle['values'].log(trial.value, step=trial._trial_id)
    447

/usr/local/lib/python3.7/dist-packages/optuna/trial/_frozen.py in value(self)
    391             if len(self._values) > 1:
    392                 raise RuntimeError(
--> 393                     "This attribute is not available during multi-objective optimization."
    394                 )
    395             return self._values[0]

RuntimeError: This attribute is not available during multi-objective optimization.

Setup
torch                         1.9.0+cu102
pytorch-lightning             1.3.8
optuna                        2.8.0
neptune-client                0.9.19
neptune-optuna                0.9.13
neptune-pytorch-lightning     0.9.7

Possible Fix
Adding study._is_multi_objective() guards to the attributes & functions mentioned in the description as in 3e1681e9dec3e4632be465e73bca8e73b059e927.

Hey @rshwndsz ,
Thank you for reporting. No words here, absolutely perfect Issue description. I really appreciate that! We'll let you know about updates :wink:

Is this issue fix? I may push a pull request @Raalsky

Hey, @patrickphat we don't made any updates yet. Feel free to contribute :wink:

would love to see this fixed :)

Stale issue - closing.

First of all, thanks for the re-open, that was a pleasant surprise for me :-)
Second, similar problems are also with the neptune.new.integrations.optuna.log_study_metadata(study, run), but from my point of view that would be probably fixed with the same fix, as it uses the same "internal" methods.
Lastly, in the meantime, for my usage, I have prepared a quick and dirty workaround (using monkey-patches) to make this work in my project, which I'm sharing with you below.
It's a bit similar to rshwndsz's awesome commit https://github.com/neptune-ai/neptune-optuna/commit/3e1681e9dec3e4632be465e73bca8e73b059e927 and doesn't solve plots, but it can be valuable for you or for anyone needing quick workaround in the meantime.
Workaround:
neptune_workarounds.py
from typing import Iterable

import neptune.new.integrations.optuna as optuna_utils
import optuna
from neptune_optuna.impl import _stringify_keys

def _workaround_log_study_details(run, study: optuna.Study):
    single_objective = not study._is_multi_objective()
    # or:  single_objective = len(study.directions) == 1

    run['study/study_name'] = study.study_name
    run['study/directions'] = study.directions
    if single_objective:
        run['study/direction'] = study.direction
    run['study/system_attrs'] = study.system_attrs
    run['study/user_attrs'] = study.user_attrs
    try:
        run['study/_study_id'] = study._study_id
        run['study/_storage'] = study._storage
    except AttributeError:
        pass

def _workaround_log_best_trials(study: optuna.Study):
    if not study.best_trials:
        return dict()

    single_objective = not study._is_multi_objective()
    # or:  single_objective = len(study.directions) == 1
    if single_objective:
        best_results = {'value': study.best_value,
                        'params': study.best_params,
                        'value|params': f'value: {study.best_value}| params: {study.best_params}'}
    else:
        best_results = {}

    for trial in study.best_trials:
        best_results[f'trials/{trial._trial_id}/datetime_start'] = trial.datetime_start
        best_results[f'trials/{trial._trial_id}/datetime_complete'] = trial.datetime_complete
        best_results[f'trials/{trial._trial_id}/duration'] = trial.duration
        best_results[f'trials/{trial._trial_id}/distributions'] = trial.distributions
        best_results[f'trials/{trial._trial_id}/intermediate_values'] = trial.intermediate_values
        best_results[f'trials/{trial._trial_id}/params'] = trial.params
        if single_objective:
            best_results[f'trials/{trial._trial_id}/value'] = trial.value
        best_results[f'trials/{trial._trial_id}/values'] = trial.values

    return best_results

def _workaround_log_trials(run, trials: Iterable[optuna.trial.FrozenTrial]):
    handle = run['trials']
    for trial in trials:
        if trial.state.is_finished() and trial.state != optuna.trial.TrialState.COMPLETE:
            handle[f'trials/{trial._trial_id}/state'] = repr(trial.state)

        single_objective = trial.values is None or len(trial.values) == 1
        if single_objective and trial.value:
            handle['values'].log(trial.value, step=trial._trial_id)

        handle['params'].log(trial.params)
        if single_objective:
            handle['values|params'].log(f'value: {trial.value}| params: {trial.params}')
        handle[f'trials/{trial._trial_id}/datetime_start'] = trial.datetime_start
        handle[f'trials/{trial._trial_id}/datetime_complete'] = trial.datetime_complete
        handle[f'trials/{trial._trial_id}/duration'] = trial.duration
        handle[f'trials/{trial._trial_id}/distributions'] = _stringify_keys(trial.distributions)
        handle[f'trials/{trial._trial_id}/intermediate_values'] = _stringify_keys(trial.intermediate_values)
        handle[f'trials/{trial._trial_id}/params'] = _stringify_keys(trial.params)
        if single_objective:
            handle[f'trials/{trial._trial_id}/value'] = trial.value
        handle[f'trials/{trial._trial_id}/values'] = trial.values

class WorkaroundNeptuneCallback(optuna_utils.NeptuneCallback):

    def _log_study_details(self, study, trial):
        if trial._trial_id == 0:
            _workaround_log_study_details(self.run, study)

    def _log_best_trials(self, study):
        self.run['best'] = _stringify_keys(_workaround_log_best_trials(study))

    def _log_trial(self, trial):
        _workaround_log_trials(self.run, [trial])

optuna_demo.py
import neptune.new as neptune
# import neptune.new.integrations.optuna as optuna_utils
import optuna
from neptune_optuna import impl as neptune_impl

from neptune_workarounds import WorkaroundNeptuneCallback, _workaround_log_study_details, _workaround_log_best_trials, \
    _workaround_log_trials

def objective(trial: optuna.Trial):
    epochs = trial.suggest_int("epochs", 3, 10)
    return 2*epochs, -epochs  # just toy example

run = neptune.init(
    project="YOUR_WORKSPACE/YOUR_PROJECT", api_token="YOUR_API_TOKEN"
)  # your credentials

params = {"directions": ["maximize", "maximize"], "n_trials": 3}
run["parameters"] = params

# neptune workarounds:
neptune_callback = WorkaroundNeptuneCallback(
    run,
    log_plot_contour=False,
    log_plot_edf=False,
    log_plot_parallel_coordinate=False,
    log_plot_param_importances=False,
    log_plot_pareto_front=False,
    log_plot_slice=False,
    log_plot_intermediate_values=False,
    log_plot_optimization_history=False,
)
# instead of:  neptune_callback = optuna_utils.NeptuneCallback(run)
# monkey patches below!
orig_log_study_details = neptune_impl._log_study_details
neptune_impl._log_study_details = _workaround_log_study_details
orig_log_best_trials = neptune_impl._log_best_trials
neptune_impl._log_best_trials = _workaround_log_best_trials
orig_log_trials = neptune_impl._log_trials
neptune_impl._log_trials = _workaround_log_trials

study = optuna.create_study(study_name="test", directions=params["directions"], load_if_exists=True)
study.optimize(objective, n_trials=params["n_trials"], catch=(Exception,), callbacks=[neptune_callback])

# you can also call log_study_metadata if you've already done the sweep/search and want to just log the params
# optuna_utils.log_study_metadata(study, run, log_plots=False)

run.stop()  # Stop Neptune logging

requirements.txt
optuna==2.10.0
plotly==5.8.0
neptune-client==0.16.3
neptune-optuna==0.9.14

@Blaizzy I'm leaving that for you to explore :)

Hi @PROrock,
Thank you for reaching out!
I will look into it and get back to you as soon as possible.

Hi  @rshwndsz, @PROrock, @patrickphat and @KMontag42!
The PR is merged, and the multi-objective support feature will be available in the next release.


Make worker asynchronous and stoppable

Changes:

Change worker API from run_forever to stop and start
Make start run worker in its own thread

@RAYemelyanova I think this this is ready now


[FEATURE] Map data type read

Support reading map from vertica

Reading map types is currently not supported. As of Vertica 11, even though Map type is defined, it is only for interoperability with external datasets and cannot be queried.


DBZ-862 Handle concurrent transaction blocking taking snapshot

https://issues.jboss.org/browse/DBZ-862

Apply after https://github.com/debezium/debezium/pull/620

@gunnarmorling This is now ready to go

@gunnarmorling https://github.com/gunnarmorling I am not a kitten
killer :-)

No, no, acknowledging is the first step on the path to forgiveness ;)

Rebased, applied and added a commit for the changes suggested above. Thanks a lot @jpechane and @jchipmunk for reviewing!


Autofixing?

Is there a means to add autofixing to composed rules?
If so, how would I go about doing that.
Specifically I'm looking to autofix no-unused-vars as reported here

There is already a package that just does that. Take a look at eslint-plugin

That was added after my original question. I would still like to know how I go about doing that so I can learn how to do so for myself

Yes, so if you go to my link, and read the code you will se a solution to an autofixer for no-unused-vars written using eslint-rule-composer which I presume was your original question?

The code technically answers the question but it doesn't walk me through it. I'm hoping for a brief tutorial or blog post on how this works. Reading code is not a substitute for teaching