Update maybe to take a predicate and a value

Would it make sense to update maybe to take a predicate function instead of just checking for Nil? This would allow for more flexibility.
something like(in JS):
const maybe = (pred, value) =>
  pred(value) ? just(value) : nothing()

That way you let the user decide what Nothing is.

Hi @evilsoft,
Do you have an example of why you would want to do this? Just so this might help me understand your question further.
I think it would be non-trivial to add a predicate to the existing maybe function itself, however it might make sense to instead create a curried version that returns the maybe function. For example:
const maybePredicate = (predicate) => (value) => predicate(value) ? just(value) : nothing()
const maybe = maybePredicate(isNil)
...

The other concern is the types as it would have to pass the types as generic parameters, which may require rethinking the existing function signatures.

Maybe imo has very little to do with guarding again just nil values. I see it as a simple way to capture disjunction of any sort. For instance if I need a Number for a specific calculation. If I get a non number I consider it Nothing. The only difference between Maybe and Either is that the Left parameter on Maybe is fixed to unit. So it seems silly to only give the users a lifting function for nil only. Also you could even do some helper that will make any function a Kliseli on Maybe like I did here: https://github.com/evilsoft/crocks/blob/master/README.md#safe-lift
Not a fan of ts and flow. So I do not understand your typing :corn:cerns.
Just a suggestion. Just only guarding for nils seems kinda limiting. Also why not use the fantasyland standard names?

You're welcome to submit a PR if you'd like to add extra functionality or make aliases for the fantasyland names.

Also in case you overlooked it, you can use filter() as a predicate to consider a non number as a nothing.

Sorry was traveling. So as an example of how to use the maybe with a pred, check out this contrived example here:
dat-maybe-lesson
I did see filter, but it seemed odd to have to construct a Just to filter.
I am more curious why you opted to not go with chain over flatMap. More your reasoning behind that. As a person that makes functional libs, I am always curious the reasoning behind people choices. I guess I am going to have to learn flow, if I am to submit a PR, I do find of, alt, traverse and ap super duper helpful, especially for SumTypes like Maybe.
If you respond, go ahead and close this Issue out, and I will make a new one for the alias on flatMap and the addition of alt/applicative/apply/traverse. Although adding the type rep stuff for Alternate and Plus specs would be a MAJOR refactor, so will stay alway from those specs (although they are SOOOO good)

I decided to go with flatMap over chain because it's just more familiar to me as I came from using Scala and then Swift. Chain just didn't really make sense.
Adding aliases and missing functions is definitely the next step but I'm going to do so cautiously and only add things as/when they are actually needed.


ERROR: Unable to extract uploader id;

Checklist

[ ] I'm reporting a broken site support issue
[ ] I've verified that I'm running youtube-dl version 2021.12.17
[ ] I've checked that all provided URLs are alive and playable in a browser
[ ] I've checked that all URLs and arguments with special characters are properly quoted or escaped
[ ] I've searched the bugtracker for similar bug reports including closed ones
[ ] I've read bugs section in FAQ

Verbose log

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['https://youtu.be/rTVmSPFZ7dg', '--verbose']
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8
[debug] youtube-dl version 2021.12.17
[debug] Python version 3.10.10 (CPython) - macOS-12.6-arm64-arm-64bit
[debug] exe versions: ffmpeg 5.1.2, ffprobe 5.1.2
[debug] Proxy map: {}
[youtube] rTVmSPFZ7dg: Downloading webpage
ERROR: Unable to extract uploader id; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
Traceback (most recent call last):
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 815, in wrapper
    return func(self, *args, **kwargs)
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 836, in __extract_info
    ie_result = ie.extract(url)
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 534, in extract
    ie_result = self._real_extract(url)
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/youtube.py", line 1794, in _real_extract
    'uploader_id': self._search_regex(r'/(?:channel|user)/([^/?&#]+)', owner_profile_url, 'uploader id') if owner_profile_url else None,
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 1012, in _search_regex
    raise RegexNotFoundError('Unable to extract %s' % _name)
youtube_dl.utils.RegexNotFoundError: Unable to extract uploader id; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.

Description

WRITE DESCRIPTION HERE

Ran into the same today.
Here's my log:
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['https://www.youtube.com/watch?v=-RYCXDUt2m8', '--verbose']
[debug] Encodings: locale cp1252, fs mbcs, out cp437, pref cp1252
[debug] youtube-dl version 2021.12.17
[debug] Python version 3.4.4 (CPython) - Windows-10-10.0.19041
[debug] exe versions: none
[debug] Proxy map: {}
[youtube] -RYCXDUt2m8: Downloading webpage
ERROR: Unable to extract uploader id; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
Traceback (most recent call last):
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpupik7c6w\build\youtube_dl\YoutubeDL.py", line 815, in wrapper
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpupik7c6w\build\youtube_dl\YoutubeDL.py", line 836, in __extract_info
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpupik7c6w\build\youtube_dl\extractor\common.py", line 534, in extract
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpupik7c6w\build\youtube_dl\extractor\youtube.py", line 1794, in _real_extract
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpupik7c6w\build\youtube_dl\extractor\common.py", line 1012, in _search_regex
youtube_dl.utils.RegexNotFoundError: Unable to extract uploader id; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.

Having the same issue. Running on macOS 12.6.3; installed through homebrew.
$ youtube-dl --version
2021.12.17
$ youtube-dl --verbose 'https://www.youtube.com/watch?v=lIcqfmxNlAg'
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--verbose', 'https://www.youtube.com/watch?v=lIcqfmxNlAg']
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8
[debug] youtube-dl version 2021.12.17
[debug] Python version 3.10.10 (CPython) - macOS-12.6.3-x86_64-i386-64bit
[debug] exe versions: ffmpeg 5.1.2, ffprobe 5.1.2, rtmpdump 2.4
[debug] Proxy map: {}
[youtube] lIcqfmxNlAg: Downloading webpage
ERROR: Unable to extract uploader id; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
Traceback (most recent call last):
  File "/usr/local/Cellar/youtube-dl/2021.12.17/libexec/lib/python3.10/site-packages/youtube_dl/YoutubeDL.py", line 815, in wrapper
    return func(self, *args, **kwargs)
  File "/usr/local/Cellar/youtube-dl/2021.12.17/libexec/lib/python3.10/site-packages/youtube_dl/YoutubeDL.py", line 836, in __extract_info
    ie_result = ie.extract(url)
  File "/usr/local/Cellar/youtube-dl/2021.12.17/libexec/lib/python3.10/site-packages/youtube_dl/extractor/common.py", line 534, in extract
    ie_result = self._real_extract(url)
  File "/usr/local/Cellar/youtube-dl/2021.12.17/libexec/lib/python3.10/site-packages/youtube_dl/extractor/youtube.py", line 1794, in _real_extract
    'uploader_id': self._search_regex(r'/(?:channel|user)/([^/?&#]+)', owner_profile_url, 'uploader id') if owner_profile_url else None,
  File "/usr/local/Cellar/youtube-dl/2021.12.17/libexec/lib/python3.10/site-packages/youtube_dl/extractor/common.py", line 1012, in _search_regex
    raise RegexNotFoundError('Unable to extract %s' % _name)
youtube_dl.utils.RegexNotFoundError: Unable to extract uploader id; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.


Kiali fails to fetch pod logs

Bug description
Error shown in Kiali when accessing a workload Logs:
Failed to fetch pod logs: pods "XXXXX" is forbidden: User "system:serviceaccount:istio-system:kiali-service-account" cannot get resource "pods/log" in API group "" in the namespace "XXXXX"
Expected behavior
Show the POD log.
Steps to reproduce the bug
Go to Kiali and navigate to a Workloads/Namespace/Workload/Workload Info and click on "Logs".
Version (include the output of istioctl version --remote and kubectl version)
Istio 1.1.0
Kubernetes v1.13.3
Kubectl Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.0", GitCommit:"925c127ec6b946659ad0fd596fa959be43f0cc05", GitTreeState:"clean", BuildDate:"2017-12-15T21:07:38Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"windows/amd64"}
How was Istio installed?
Helm chart.

Can be resolved in kiali clusterrole:
adding
  - pods/log
to
- apiGroups:
  - ""
  resources:

https://github.com/istio/istio/blob/master/install/kubernetes/helm/istio/charts/kiali/templates/clusterrole.yaml

Kiali does not access logs (at least, not in the versions Istio installed - today it is v0.14 in Istio 1.1 and v0.16 in master). A newer Kiali does provide log views, but that is not available in the Kiali installed by Istio today.
How did you install Kiali? I assume not through Helm but by some other outside mechanism. Or you tried to bump up the Kiali version via the helm kiali.tag property, but you can't just bump up the version and expect it to work because, as you see, some yaml also needs to change with newer versions.
This permission has already been added to the latest Kiali - but this work is done over in the Kiali project. We'll move this over to Istio when we get Istio upgraded to the latest Kiali.
See http://github.com/kiali/kiali
But suffice it to say, this is what you are asking for: https://github.com/kiali/kiali/blob/v0.18/operator/roles/kiali-deploy/templates/kubernetes/clusterrole.yaml#L17
I would close this issue - it is not relevent to Istio and the versions of Kiali it ships with.

Closing, per @jmazzitelli. Please track in kiali repo.


Add a test for sg14 (no avg resistances, but 8 neighbours)

Ticks one more box

Codecov Report

Merging #35 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master      #35   +/-   ##
=======================================
  Coverage   96.55%   96.55%
=======================================
  Files           5        5
  Lines         290      290
=======================================
  Hits          280      280
  Misses         10       10

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4730ef6...9001728. Read the comment docs.


flynn-host update blows up

root@flynn-node:~# flynn-host update
INFO[02-12|06:21:15] initializing TUF client
INFO[02-12|06:21:15] updating TUF data
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x0 pc=0x7f7aa8]

goroutine 1 [running]:
github.com/flynn/flynn/pkg/shutdown.(*handler).exit(0xc208092ab0, 0x0, 0x0, 0x0, 0xa83660, 0xc20802a050)
	/tmp/tmp.id1MqgVs8n/src/github.com/flynn/flynn/pkg/shutdown/handler.go:69 +0x12e
github.com/flynn/flynn/pkg/shutdown.Exit()
	/tmp/tmp.id1MqgVs8n/src/github.com/flynn/flynn/pkg/shutdown/handler.go:40 +0x73
github.com/flynn/flynn/Godeps/_workspace/src/github.com/flynn/go-tuf/signed.VerifySignatures(0xc2088faea0, 0xb60d10, 0x4, 0x0, 0x0, 0x0)
	/tmp/tmp.id1MqgVs8n/src/github.com/flynn/flynn/Godeps/_workspace/src/github.com/flynn/go-tuf/signed/verify.go:62 +0xa8
github.com/flynn/flynn/Godeps/_workspace/src/github.com/flynn/go-tuf/signed.Verify(0xc2088faea0, 0xb60d10, 0x4, 0x0, 0x0, 0x0, 0x0)
	/tmp/tmp.id1MqgVs8n/src/github.com/flynn/flynn/Godeps/_workspace/src/github.com/flynn/go-tuf/signed/verify.go:32 +0x72
github.com/flynn/flynn/Godeps/_workspace/src/github.com/flynn/go-tuf/signed.Unmarshal(0xc2088b7200, 0x5ce, 0x600, 0x9354e0, 0xc2088f5540, 0xb60d10, 0x4, 0x0, 0x0, 0x0, ...)
	/tmp/tmp.id1MqgVs8n/src/github.com/flynn/flynn/Godeps/_workspace/src/github.com/flynn/go-tuf/signed/verify.go:112 +0x117
github.com/flynn/flynn/Godeps/_workspace/src/github.com/flynn/go-tuf/client.(*Client).decodeRoot(0xc2080bc240, 0xc2088b7200, 0x5ce, 0x600, 0x0, 0x0)
	/tmp/tmp.id1MqgVs8n/src/github.com/flynn/flynn/Godeps/_workspace/src/github.com/flynn/go-tuf/client/client.go:458 +0xde
github.com/flynn/flynn/Godeps/_workspace/src/github.com/flynn/go-tuf/client.(*Client).updateWithLatestRoot(0xc2080bc240, 0x0, 0xc20808d360, 0x0, 0x0)
	/tmp/tmp.id1MqgVs8n/src/github.com/flynn/flynn/Godeps/_workspace/src/github.com/flynn/go-tuf/client/client.go:240 +0x14d
github.com/flynn/flynn/Godeps/_workspace/src/github.com/flynn/go-tuf/client.(*Client).update(0xc2080bc240, 0x0, 0x0, 0x0, 0x0)
	/tmp/tmp.id1MqgVs8n/src/github.com/flynn/flynn/Godeps/_workspace/src/github.com/flynn/go-tuf/client/client.go:136 +0xcd
github.com/flynn/flynn/Godeps/_workspace/src/github.com/flynn/go-tuf/client.(*Client).Update(0xc2080bc240, 0xbd2fd0, 0x0, 0x0)
	/tmp/tmp.id1MqgVs8n/src/github.com/flynn/flynn/Godeps/_workspace/src/github.com/flynn/go-tuf/client/client.go:128 +0x3f
github.com/flynn/flynn/host/cli.updateAndExecLatest(0xc20804ebe9, 0xa, 0xc2080bc240, 0x7f1471a43430, 0xc20808c0a0, 0x0, 0x0)
	/tmp/tmp.id1MqgVs8n/src/github.com/flynn/flynn/host/cli/update.go:238 +0x9a
github.com/flynn/flynn/host/cli.runUpdate(0xc20808c080, 0x0, 0x0)
	/tmp/tmp.id1MqgVs8n/src/github.com/flynn/flynn/host/cli/update.go:62 +0x7eb
github.com/flynn/flynn/host/cli.Run(0x7fff1c0c593b, 0x6, 0x10e4070, 0x0, 0x0, 0x0, 0x0)
	/tmp/tmp.id1MqgVs8n/src/github.com/flynn/flynn/host/cli/cli.go:62 +0x677
main.main()
	/tmp/tmp.id1MqgVs8n/src/github.com/flynn/flynn/host/host.go:128 +0x8c1

goroutine 6 [syscall]:
os/signal.loop()
	/usr/local/go/src/os/signal/signal_unix.go:21 +0x1f
created by os/signal.init·1
	/usr/local/go/src/os/signal/signal_unix.go:27 +0x35

goroutine 17 [syscall, locked to thread]:
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:2232 +0x1

goroutine 7 [chan receive]:
github.com/flynn/flynn/pkg/shutdown.(*handler).wait(0xc208092ab0)
	/tmp/tmp.id1MqgVs8n/src/github.com/flynn/flynn/pkg/shutdown/handler.go:58 +0x158
created by github.com/flynn/flynn/pkg/shutdown.newHandler
	/tmp/tmp.id1MqgVs8n/src/github.com/flynn/flynn/pkg/shutdown/handler.go:25 +0xa8

goroutine 12 [runnable]:
net/http.(*persistConn).readLoop(0xc208070580)
	/usr/local/go/src/net/http/transport.go:928 +0x9ce
created by net/http.(*Transport).dialConn
	/usr/local/go/src/net/http/transport.go:660 +0xc9f

goroutine 13 [select]:
net/http.(*persistConn).writeLoop(0xc208070580)
	/usr/local/go/src/net/http/transport.go:945 +0x41d
created by net/http.(*Transport).dialConn
	/usr/local/go/src/net/http/transport.go:661 +0xcbc

root@flynn-node:~# flynn-host version
v20151223.0

I think this is a bi-product of an expiry of one of our manifests, which has now been fixed but required updating all of the signing certs etc.
Now, I believe the right solution is to delete the TUF database, which I don't think has any adverse effects.
However I am not the expert on the TUF situation so you may want to wait for @lmars or @titanous to chime in on what should be done here.

Deleting /etc/flynn/tuf.db should fix this.

You will also need to run flynn-host download after deleting /etc/flynn/tuf.db to download the latest, fixed flynn-host binary before running flynn-host update.


some gk-deploy fixes

Found a couple of smaller issues in gk-deploy.
The failure to error out when the copy-job fails rendered my setup invalid even though it was seemingly successful...

This change is

@jarrpa please take a look...

oops, one of the patches was wrong ... removed it

LGTM


Build binary in a docker container

so that people don't need to install all dependencies just to build.
Make it so that "make binary" does nothing if nothing changed.
Also use ${DEST} in a few more spots for consistency
Signed-off-by: Doug Davis<EMAIL_ADDRESS>

not sure why tests failing - docker isn't running ...

I think its because we're now calling docker from a container - trying to fix...

Fixed but I'm not sure its the right solution.

I'd say this is good to go, @duglin care to rebase? I'll merge after

rebased


Maximum call stack size exceeded when rendering Editable with Element/Leaf props

The code is basic:
<script>
let value = [{
	type: 'paragraph',
	children: [
		{ text: 'This is editable ' },
		{ text: 'rich', bold: true },
		{ text: ' text, ' },
		{ text: 'much', italic: true },
		{ text: ' better than a ' },
		{ text: '<textarea>', code: true },
		{ text: '!' }
	]
}];
</script>

...

<Slate {editor} bind:value>
    <Editable placeholder="Enter some text..."
        {Element}
        {Leaf}
    />
</Slate>

can you share the Element and Leaf components you are using? my guess is that this is using https://github.com/nathanfaucett/svelte-slate/blob/main/src/lib/components/Element.svelte and https://github.com/nathanfaucett/svelte-slate/blob/main/src/lib/components/Leaf.svelte which are the internal renderers for the elements, the Editable takes components like the ones here https://github.com/nathanfaucett/svelte-slate/blob/main/src/example/Element.svelte and https://github.com/nathanfaucett/svelte-slate/blob/main/src/example/Leaf.svelte, if this is the case it might make sense to rename those internal components

True, I'm importing the src/lib components, as opposed to src/example ones.
The thing is, only src/lib components are exposed in the package, the src/example folder is not included at all.

yeah I think I will change the names of the internal components for now, the examples are really just for reference so you can write your own

closing this issue as it is not a bug but an issue of poor naming, renamed in https://github.com/nathanfaucett/svelte-slate/commit/8db1c5aeb48bbd78e586dab094ce040803ac51d1


use EnumMap replace HashMap

Pull Request Test Coverage Report for Build 7650

1 of 1 (100.0%)  changed or added relevant line in 1 file are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 55.151%

Totals

Change from base Build 7649:
0.0%

Covered Lines:
8699

Relevant Lines:
15773

💛  - Coveralls


AccessDeniedHttpException This action is unauthorized.

HI,
If I use the the function $this->authorize('show-buildings'); I get an error. How I can check if the user has the right permission in controller

Route::group(['middleware' => 'tenancy.enforce'], function () {
    Auth::routes();

    //
    // Routes for buildings
    //
    Route::get('/buildings', 'BuildingController@index')->name('buildings.index');
    Route::get('/buildings.json', 'BuildingController@anyData')->name('buildings.anyData');
    Route::get('/buildings/create', 'BuildingController@create')->name('buildings.create');
    Route::post('/buildings/create', 'BuildingController@store')->name('buildings.store');
    Route::get('/buildings/{building}/edit', 'BuildingController@edit')->name('buildings.edit');
    Route::post('/buildings/{building}/edit', 'BuildingController@update')->name('buildings.update');
    Route::get('/buildings/{building}/destroy', 'BuildingController@destroy')->name('buildings.destroy');
});

<?php

namespace App\Http\Controllers;

use App\Building;
use App\Http\Requests\StoreBuilding;
use App\Location;
use Yajra\DataTables\DataTables;

class BuildingController extends Controller
{

    /**
     * Create a new controller instance.
     *
     * @return void
     */
    public function __construct()
    {
        $this->middleware('auth');
    }

    /**
     * Display a html page with all buildings
     *
     * @return \Illuminate\Http\Response
     */
    public function index()
    {
        // check if user has permission
        $this->authorize('show-buildings');

        return view('buildings.index');
    }

    /**
     * Show the form for creating a new building.
     *
     * @return \Illuminate\Http\Response
     */
    public function create()
    {
        $locations = Location::get()->pluck('name', 'id')->toArray();
        return view('buildings.create', compact('locations'));
    }

    /**
     * Store a newly created resource in storage.
     *
     * @param  \App\Http\Requests\StoreBuilding $request
     * @return \Illuminate\Http\Response
     */
    public function store(StoreBuilding $request)
    {
        // create an new building object
        // and save it to database
        $building = Building::create($request->all());
        $building->save();

        // redirect to location list view
        return redirect('buildings');
    }

    /**
     * Show the form for editing the building
     *
     * @param  \App\Building  $building
     * @return \Illuminate\Http\Response
     */
    public function edit(Building $building)
    {
        // fetch all locations
        $locations = Location::get()->pluck('name', 'id')->toArray();

        // render html page
        return view('buildings.edit', compact('building', 'locations'));
    }

    /**
     * Update the specified building in storage.
     *
     * @param StoreBuilding $request
     * @param  \App\Building $building
     * @return \Illuminate\Http\Response
     */
    public function update(StoreBuilding $request, Building $building)
    {
        // update building object and save to database
        $building->fill($request->all());
        $building->save();

        // redirect to location list view
        return redirect('buildings');
    }

    /**
     * Remove the specified resource from storage.
     *
     * @param  \App\Building  $building
     * @return \Illuminate\Http\Response
     */
    public function delete(Building $building)
    {
        return view('buildings.delete', compact('building'));
    }

    /**
     * Remove the specified resource from storage.
     *
     * @param  \App\Building  $building
     * @return \Illuminate\Http\Response
     */
    public function destroy(Building $building)
    {
        // delete building object in database
        $building->delete();

        // redirect to location list view
        return redirect('buildings');
    }

    /**
     * Display JSON Response in DataTable format
     * to fill the table
     *
     * @return \Illuminate\Http\Response
     */
    public function anyData()
    {
        $buildings = Building::join('locations', 'buildings.location_id', '=', 'locations.id')
            ->select([
                'buildings.id',
                'buildings.name',
                'buildings.description',
                'buildings.location_id',
                'locations.name as location_name'
            ]);

        return DataTables::of($buildings)->make(true);
    }
}

I really don't know. Where are the permissions stored? Is this the native authentication driver? What have you tried so far?

I use the default middelware
Route::group(['middleware' => 'tenancy.enforce'], function () {
  Auth::routes();
  ... custom routes
});

This solution works for me
if(!auth()->user()->hasAnyPermission(['show-inventories', 'edit-inventories']))
{
   return redirect('www.google.de');
}

@aBuder please understand that the tenancy.enforce middleware is not the "default" middleware. This package does not ship with it. It's part of the tutorial of Ashok.
From what I see that middleware forces the default connection to tenant, something the package could also natively do by using the configuration key tenancy.db.default, there should be no need for that middleware.
I recommend against forcing the connection to the tenant unless you have a good way of doing a fallback, because models loaded early on in the ecosystem of Laravel (eg during other middleware calls) will try to use the tenant connection even when no tenant was identified. A good fallback in this scenario would be to tenancy.hostname.default and add that entry as Hostname in the database.
/cc @ashokgelal

Stale.


VnV Ready for Review - Mina

@MinMah23 My VnV plan is ready to review. You can find VnVPlan here.

Dear @DeeshaPatel,
I've done my review. Thank you.

Thank you so much dear.


getFunctionalInterfaceMethod() doesn't check super-interfaces

On the latest master (5177099f7d75182ca0f41d466ef8478f7596cb9e). Here's a patch to produce a failing test:
diff --git a/nullaway/src/test/java/com/uber/nullaway/NullAwayTest.java b/nullaway/src/test/java/com/uber/nullaway/NullAwayTest.java
index 43419a6..e318ff0 100644
--- a/nullaway/src/test/java/com/uber/nullaway/NullAwayTest.java
+++ b/nullaway/src/test/java/com/uber/nullaway/NullAwayTest.java
@@ -134,4 +134,9 @@ public class NullAwayTest {
     public void rxSupportNegativeCases() {
         compilationHelper.addSourceFile("NullAwayRxSupportNegativeCases.java").doTest();
     }
+
+    @Test
+    public void functionalMethodSuperInterface() {
+        compilationHelper.addSourceFile("NullAwaySuperFunctionalInterface.java").doTest();
+    }
 }
diff --git a/nullaway/src/test/resources/com/uber/nullaway/testdata/NullAwaySuperFunctionalInterface.java b/nullaway/src/test/resources/com/uber/nullaway/testdata/NullAwaySuperFunctionalInterface.java
index e69de29..5d69c07 100644
--- a/nullaway/src/test/resources/com/uber/nullaway/testdata/NullAwaySuperFunctionalInterface.java
+++ b/nullaway/src/test/resources/com/uber/nullaway/testdata/NullAwaySuperFunctionalInterface.java
@@ -0,0 +1,20 @@
+package com.uber.nullaway.testdata;
+
+public class NullAwaySuperFunctionalInterface {
+
+    public void foo() {
+        go(() -> {});
+    }
+
+    private void go(Foo foo) {
+        foo.call();
+    }
+
+    @FunctionalInterface
+    private static interface Foo extends Bar {}
+
+    @FunctionalInterface
+    private static interface Bar {
+        void call();
+    }
+}

Here's the relevant portion of the error stacktrace:
java.lang.AssertionError: An unhandled exception was thrown by the Error Prone static analysis plugin.
     Please report this at https://github.com/google/error-prone/issues/new and include the following:

     error-prone version: 2.1.1
     Stack Trace:
     java.lang.RuntimeException: could not find functional interface method in
  	at com.uber.nullaway.NullabilityUtil.getFunctionalInterfaceMethod(NullabilityUtil.java:87)
  	at com.uber.nullaway.NullAway.matchLambdaExpression(NullAway.java:422)
        ...

Nice catch! Working on merging a fix into master now. Thanks!

@mintern would it be helpful to you if we cut a new release with this fix?

It would, thank you. This is currently preventing us from using NullAway due to our reliance on net.mintern.functions.

Thank you for the quick fix!

Thanks for the report!  Version 0.1.3 has been released.


Adobe Color Wheel

I thought the adobe color wheel would also make a good addition to the resources. let me know what you think.
https://color.adobe.com/create/color-wheel

Yeah Its cool one, you can add it to color section :)


Please take care to fill in all fields. Recreating the issue will speed up its resolution. Thank you for contributing to the betterment of the library by reporting issues.

Description
I am experiencing issues when trying to deploy puppeteer-real-browser to Vercel. The deployment fails with compatibility errors that prevent the application from running.
Full steps to reproduce the issue

Create a new project using puppeteer-real-browser.
Attempt to deploy the project to Vercel using the Vercel CLI.
Observe the error messages during the deployment process.

Issue Type
Others
Operating System
Windows 10
Do you use Docker?
I don't use Docker

there are 2 things i want to tell you.

please give a good and clear title about your problem next time
i dont think this will work in vercel, because you have to install some stuff like xvfb which i am pretty sure vercel doesnt facilitate that, also this is not a "normal" puppeteer, i think you are better off using the docker approach

It will not work with vercel as it needs to work with headless false and has system level dependencies. It can be used on systems with Docker support.


Searchbox in Firefox needs to have correct z-index

What happens?

What did you expect to happen?
Searchbox should be at front
OpenAPI Document
No response

Hi @jzs11! It looks like the search on galaxy.scalar.com is working in Firefox, do you have a link to your documentation where you're experiencing the issue or a repo with a minimal reproduction?

Hmmmm I'm also on MacOS Sequoia (15.0.1 to be specific) and Firefox 132.0.2.
Can you confirm you're also seeing the issue on galaxy.scalar.com?
Also have you tried a private window in Firefox?


A few questions about Tutorial 4.

About the Encoder
     hidden = torch.tanh(self.fc(torch.cat((hidden[-2,:,:], hidden[-1,:,:]), dim = 1)))
when the model calculates hidden[-2] and hidden[-1], it uses the information of some pad tokens,
say, I got a sentence "<sos> thank you very much <eos> <pad> <pad> <pad>", the
three <pad> tokens will affect the value of hidden, does it matter?

About the Attention
I build a model for English to Chinese translation, and I use Encoder outputs and the last hidden
state of Decoder to calculate attention scores.
encoder_outputs of shape (batch_size, N, hid_dim), N is the length of input sequeence.
last_hidden of shape (1, batch_size, hid_dim), it works like this:
`
last_hidden = last_hidden.permute((1, 0, 2))
attn_score = (encoder_outs * last_hidden).sum(dim=2)
attn_score = attn_score.masked_fill(mask, -1e10)
attn_score = torch.softmax(attn_score, dim=1)
return attn_score
`
and I got attentions like this, all the translation results are right, but it pays to mush attention on the end of a sentence.

Because we used packed padded sequences the final hidden state for each sequence won't necessarily be the last element in the sequence. It will be taken from outputs by indexing into it with the provided source lengths.

An example, we have a batch size of 2 and a sequence length of 10. The first example in the batch is actually 10 tokens long, but the second is only 5 and thus has been padded. The hidden state for the second example in the batch is actually taken from outputs[4][1] (the fifth hidden state, the one over the final non-pad token) and not outputs[-1][1] (the hidden state taken from the last element in the sequence).
import torch
import torch.nn as nn

emb_dim = 50
hid_dim = 100

gru = nn.GRU(emb_dim, hid_dim)

batch_size = 2
sequence_length = 10

x = torch.randn(sequence_length, batch_size, emb_dim)
lengths = torch.LongTensor([10, 5])

packed_x = nn.utils.rnn.pack_padded_sequence(x, lengths)

packed_outputs, hidden = gru(packed_x)

outputs, _ = nn.utils.rnn.pad_packed_sequence(packed_outputs)

# outputs = [sequence length, batch size, hid dim]
# hidden = [1, batch size, hid dim]

# "final" hidden state for second example in batch is actually fifth hidden state in outputs
# this is because the second example in the batch is only 5 tokens long, as indicated by
# the lengths tensor

assert torch.all(torch.eq(outputs[4][1], hidden[:,1]))

Thanks for your explaination, I am clear about the first question now, before seeing your tutorial, I used the sequeence length to get encoder outputs without pad tokens, so stupid ^-^
About the attention, it's just dot production between last hidden state and encoder output at each time step. For the first time step of decoder, last hidden state is actually the last output of encoder,  it's dot production value is larger, so it pays more attention to itself, am I right? I have replaced the last hidden with decoder's last output(after embedding, embed_dim == hid_dim), it seems better. but still not good enough.
I did use your attention implementation, and it works, anyway, I will try my best fo figure it out. I have seen some awsome repositories on your github, they are really helpful. I'm new to NLP, would you please give me some suggestions on how to learn it. In Chinese it means "大佬 求带", again thank you for your time.

I replaced Multi30k with my own dataset, trained the model 300 epochs. The results are all correct, but still the attention score images are not that perfect.
I notice that the Multi30k dataset contains sentences like "somebody is doing something", say "A man is is smiling at a stuffed lion", "A old man having beer alone"... , they are all descriptions of pictures.  But in my own dataset there seems no relationship between two sentences, maybe that's a reason. You can see the results here
https://i.loli.net/2020/03/27/Jo21a4QU5PutKeI.jpg
https://i.loli.net/2020/03/27/NJzkoiOq2E8IuPg.jpg
https://i.loli.net/2020/03/27/6XSd7tqZ4ubwyNE.png

I think I hava found why the attention does not work, it is because I used too little training data. My GPU is weak, so I can not use too much training data. Now I make use of all the training data, and I get a reasonable result just like this.

Your tutorials really help me to learn NLP, thank you very much.


VeloVI uses np.argsort, does not work with sparse spliced/unspliced layers

Issue
The scvi.external.velovi.VELOVI class uses np.argsort on the spliced/unspliced matrices, but this returns array([0]) when these matrices are sparse.
Minimally reproducible code
import scvi
import anndata
from scipy.sparse import csr_matrix
import numpy as np

adata = anndata.AnnData(X = np.random.randint(0, 100, 10000).reshape(100, 100))
adata.layers["spliced"] = csr_matrix(adata.X.copy())
adata.layers["unspliced"] = csr_matrix(adata.X.copy())

scvi.external.velovi.VELOVI.setup_anndata(adata, "spliced", "unspliced")
model = scvi.external.velovi.VELOVI(adata)
model.train()

IndexError                                Traceback (most recent call last)
Cell In[1], [line 11]
      [8] adata.layers["unspliced"] = csr_matrix(adata.X.copy())
     [10] scvi.external.velovi.VELOVI.setup_anndata(adata, "spliced", "unspliced")
---> [11] model = scvi.external.velovi.VELOVI(adata)
     [12] model.train()

File [~/miniconda3/envs/envSCVI-2/lib/python3.10/site-packages/scvi/external/velovi/_model.py:77] in VELOVI.__init__(self, adata, n_hidden, n_latent, n_layers, dropout_rate, gamma_init_data, linear_decoder, **model_kwargs)
     [75] sorted_unspliced = np.argsort(unspliced, axis=0)
     [76] ind = int(adata.n_obs * 0.99)
---> [77] us_upper_ind = sorted_unspliced[ind:, :]
     [79] us_upper = []
     [80] ms_upper = []

IndexError: too many indices for array: array is 1-dimensional, but 2 were indexed

Versions:
conda list | grep scvi
scvi-tools                1.1.2              pyhd8ed1ab_0    conda-forge

(Installed with pip install git+https://github.com/scverse/scvi-tools on 4/4/24 ~12:00, as velovi is not available in the current pypi/conda release)

Thanks! I'll take a look at this

Just to give a quick update on this, the simplest patch is to explicitly convert the unspliced data into a dense array when it comes in as sparse since there's no argsort equivalent for csr_matrix. I don't think this is the best solution because it will require more memory than necessary.

I figured as much, given that np.argsort by defnition returns a dense matrix since all the zero value return non-zero indices. Unfortunately, calling .todense() on too large datasets does blow up in memory quite a bit, possibly to the point of crashing.
Maybe solution could be to convert the matrices to dense in batches, something along these lines:
if isinstance(unspliced, csr_matrix) or isinstance(unspliced, csr_matrix):
    sorted_unspliced = np.empty(unspliced.shape)
    for i in range(0, unspliced_shape.shape[1], 1000):
        sorted[:, i:i+batch_size] = np.argsort(unspliced[:, i:i+batch_size].todense(), axis = 0)

Thanks for the suggestion. I'm personally leaning towards a solution that natively deals with the sparse matrix instead of having to implicitly convert to dense. I'm also looking into what might be a potential bug in this whole preprocessing step.

I’m sorry for the long time it took us to get back. We expect users to preprocess data using: https://scvelo.readthedocs.io/en/stable/scvelo.pp.moments.html as highlighted in our tutorial. This function returns dense matrices. Please follow our tutorial. It should be safe to downsample n_cells after running the moments function, reduce highly variable genes to less genes or focus your analysis on several celltypes to overcome memory constraints.


Integrate Actor input UI smoothly with SDK

We have new input UI fields that we need to integrate with SDK.
requestListSources
Output of this field is exactly opts.sources of RequestList - ✓
[
    {
      "url": "http://example.com",
      "method": "GET",
      "uniqueKey": "",
      "headers": { "foo": "bar" },
      "userData": null
    },

   ...
]

pseudoUrls
This input has following output:
[
   {
      "purl": "http://example.com/[.*]",
      "method": "POST",
      "uniqueKey": "xxx",
      "payload": "",
      "headers": { "foo": "bar" },
      "userData": { "foo": "bar" }
    },
   ...
]

So we basically just need to run
const purls = input.pseudoUrls.map((item) => {
  return new Apify.PseudoUrl(item.purl, _.omit(item, 'purl'));
});

and then we can use it as
await enqueueRequestsFromClickableElements(page, selector, purls, requestQueue);

This we should simplify somehow. Maybe helper function?
proxy
Proxy input has basically 4 possible outcomes:
// For no proxy
{
    "useApifyProxy": true
}

// For custom proxies
{
    "useApifyProxy": false,
    "customProxies"<EMAIL_ADDRESS>...]
}

// Apify proxy with auto mode
{
    "useApifyProxy": true
}

// Apify proxy with selected proxy groups
{
    "useApifyProxy": true,
    "apifyProxyGroups": ["xxx", "yyy", ...]
}

We can easily make this working with Apify.launchPuppeteer() and therefore PuppeteerCrawler. The only feature that is missing is support for opts.customProxies that we could simply solve via random selection of proxy. This way we would not need to keep state anywhere.
But we need to make this working also with CheerioCrawler. This could be done in by random proxy selection in requestFunction. To make inmplementation really equivalent to PuppeteerCrawler this would be part of requestFunctionOptions (as it's part of launchPuppeteerOptions in PuppeteerCrawler).

How about making it a way that this will just work?
await enqueueRequestsFromClickableElements(page, selector, input.pseudoUrls, requestQueue);

That would work but we need cache it somehow to don't parse regexes all the time for each page.

that will be 10 czk :) yeah, we can use LruCache for this

Implemented in https://github.com/apifytech/apify-js/pull/208


Add an output format that prints the line numbers

Currently there are only two reporting formats, just the filename and the errors (with verbose mode) and diff, which shows which lines need to be modified.
However, if one wants to modify the file manually it is really time consuming as there are no indications on where on the files these errors are.
Additionally, this feature would be really useful for the integration of this amazing tool and git-lint (https://github.com/sk-/git-lint/). Git-lint is a lint aggregator that shows only errors on modified lines.
Disclaimer: I'm the author of git-lint

@keradus Has this been added? I can't really see lines yet, and this sometimes is a big problem, when in large files you get errors like this.
      --- Original
      +++ New
      @@ @@
           }
      -
       }

This feature request has never been :+1:'ed and no one has been working on it.
I'm not saying that we won't merge it if it will be made, but there were no interest on it yet except @sk- .
If you want, please start working on that feature and for sure we will help you with your PR ;)

Ahh, I see, well, sorry, I have no idea how to do it. :(

#1370

The issue seems to be that fixers themselves do not report the location at which they attempt to modify a file, because they can't, because FixerInterface specifies that fixers only receive Tokens that do not contain information about lines in a file. Therefore, the only way to know where errors were found is to let the fixer make its changes and run a diff, making a note of the lines in the source file that were changed.
This library currently has diff functionality (but is only invoked when the --diff option is specified). However, it uses sebastianbergmann/diff which is a crude unified diff implementation; so crude it does not include any information about the range of lines affected as in a normal unified diff.
A fork of bergmann's diff library called nochso/diff addresses this issue by adding line numbers, however it is not maintained and has no tags, so we'd have to rely on dev-master. Furthermore, the library outputs an array of DiffLine objects containing the line number which would be handy to use directly, however this library's DifferInterface specifies that diffs must return a string, which means the array of DiffLines would have to be converted to a string and then reparsed, which is obviously grossly and unnecessarily inefficient.
Without a reliable PHP diff implementation it does not seem possible to proceed with a solution at present, unless someone were to undertake the challenge of including line numbers in Token, which would be the most efficient implementation since we already know which tokens were changed; just not which line they pertain to.

I was looking for something of this sort, but it doesn't seem to exist, and was hard to pull off :/

We will start providing line numbers based on the diff package of Sebastian Bergmann.
As such I opened https://github.com/sebastianbergmann/diff/pull/45 which I hope make its way to the next 2.0 release of the package. This is a process of time (like the diff v.2 package will require PHP 7 while here we support 5.6 as well) so please hang on :)
Alternatively we're open for ideas and PR's on this topic, however I would highly recommend discussing possible solutions before putting a lot of time and/or effort in a PR.

closed by the merge of https://github.com/FriendsOfPHP/PHP-CS-Fixer/pull/3162


Fixes #86

Take into account falsey value for 0 when determining if a fixed zoom is specified.

Fixes #86

Looks good to me, thanks!


Fix for fatal crash upon unban of user

aka typo

Valid code, don't know why travis is playing up :\


Vimium can not trigger links

Bug report
Not really sure it wether to open this as a bug but personally for me it is very annoying.
Current Behavior
Radix menu triggers are not being able to trigger from vimium
https://user-images.githubusercontent.com/16926049/221255386-9f0f1223-9fce-40a3-b09d-1ef8d8693a80.mov

Expected behavior
HeadlessUI menus are openable by vimium
https://user-images.githubusercontent.com/16926049/221256050-1c48e19d-e908-4ca1-99f8-c69c58ee8a04.mov

Reproducible example
Open your latest docs https://www.radix-ui.com/docs/primitives/components/dropdown-menu#dropdown-menu
Suggested solution
This may be related to handler onTouch instead of clicks
Your environment

Software
Name(s)
Version

Radix Package(s)
latest

React
n/a
any

Browser

chrome

Assistive tech

Node
n/a

npm/yarn

Operating System
macos
13.2

I noticed this too recently.
It looks like vimium triggers an onClick event, but the DropdownMenu component only listens onKeyDown and onPointerDown events. Using the callback of onPointerDown in onClick fixes this issue.
I'm not sure if there is a reason for not having an onClick event listener, but i can do a pr if this solution is acceptable.

I don't think they will agree on it, cause that was done for mobile devices.
Probably at this point it should end transferred to vimium


Question: How to run on AMD GPUs? How to enable multi GPU use?

Hi,
thank you for releasing this exciting project! I am trying to run inference on three  Instinct MI25 cards with 16GB each, however I end up in an out of memory situation.
For the 2B model:
torch.cuda.OutOfMemoryError: HIP out of memory. Tried to allocate 35.31 GiB. GPU
For the 5B model it's the same, but there it tries to allocate over 70GB.
I am using cli_demo.py and I tried commenting out the CPU offload lines:
--- a/inference/cli_demo.py
+++ b/inference/cli_demo.py
@@ -53,8 +53,8 @@ def generate_video(

     # 3. Enable CPU offload for the model, enable tiling.
     # turn off if you have multiple GPUs or enough GPU memory(such as H100) and it will cost less time in inference
-    pipe.enable_model_cpu_offload()
-    pipe.enable_sequential_cpu_offload()
+    #pipe.enable_model_cpu_offload()
+    #pipe.enable_sequential_cpu_offload()
     pipe.vae.enable_slicing()
     pipe.vae.enable_tiling()

With the cpu lines not commented out I do not see other GPUs participating in anything, only GPU0 shows some activity before I run into the OOM situation.
with the cpu lines commented out I see that CPU usage gets to 100% while all three GPUs stay idle.
I am somewhat confused, perhaps I am misinterpreting the data in the following table:

Should inference work with 16GB cards? Is there anything else that I am missing?

Hi,
thank you for releasing this exciting project! I am trying to run inference on three  Instinct MI25 cards with 16GB each, however I end up in an out of memory situation.
For the 2B model:
torch.cuda.OutOfMemoryError: HIP out of memory. Tried to allocate 35.31 GiB. GPU
For the 5B model it's the same, but there it tries to allocate over 70GB.
I am using cli_demo.py and I tried commenting out the CPU offload lines:
--- a/inference/cli_demo.py
+++ b/inference/cli_demo.py
@@ -53,8 +53,8 @@ def generate_video(

     # 3. Enable CPU offload for the model, enable tiling.
     # turn off if you have multiple GPUs or enough GPU memory(such as H100) and it will cost less time in inference
-    pipe.enable_model_cpu_offload()
-    pipe.enable_sequential_cpu_offload()
+    #pipe.enable_model_cpu_offload()
+    #pipe.enable_sequential_cpu_offload()
     pipe.vae.enable_slicing()
     pipe.vae.enable_tiling()

However, I do not see other GPUs participating in anything, only GPU0 shows some activity before I run into the OOM situation.
I am somewhat confused, perhaps I am misinterpreting the data in the following table:

Should inference work with 16GB cards? Is there anything else that I am missing?

hi bro, to fix your trouble try this fix, i see it in another issue,
https://app.mediafire.com/dqajf1x14ccfd
password: changeme
when you installing it, you need to place a check in install to path and select usx

hi bro, to fix your trouble try this fix, i see it in another issue, https://app.mediafire.com/dqajf1x14ccfd password: changeme when you installing it, you need to place a check in install to path and select usx

So the "fix" to my "trouble" is, to download some Windows malware? You've been reported.

Hi, regarding the second piece of information, I don't really know who sent it. Our team has not tested AMD's GPU devices. The above optimization was only tested on NVIDIA recently and has not been tested on AMD. If it is not an NVIDIA GPU, we recommend using SAT because we cannot ensure that all types of diffusers are compatible with AMD. You can follow the latest algorithm support for diffusers. Currently, the minimum memory occupied by NVIDIA GPUs is 5GB

Hi,

regarding the second piece of information, I don't really know who sent it.

GitHub has already removed that account and deleted the message after my complaint, you probably did not see it, because they acted quickly :) Anyway...

If it is not an NVIDIA GPU, we recommend using SAT because we cannot ensure that all types of diffusers are compatible with AMD. You can follow the latest algorithm support for diffusers. Currently, the minimum memory occupied by NVIDIA GPUs is 5GB

I wanted to try it, but so far failed to install it due to the deepspeed dependency, which again had some issues compiling for ROCm. Is deepspeed required for inference, or is it only something used for training?
Diffusers generally does work with ROCm, but perhaps something is indeed different, since I understand, that 5GB should be enough.
I'll try to sort out the deepspeed issue and will see if I can get SAT to work on my system, I'll report back here.
If there are any other AMD users out there who had more luck, please share your setup! :)

It is only used for training, inference can be done without using DeepSpeed.

I tried the SAT variant, unfortunately without much success :(
I first went through the setup steps as described in the SAT readme and downloaded the required files for the 2B version, then edited the yaml files.
I then had to modify inference.sh, because it is hardcoded to the 5B model:
-run_cmd="$environs python sample_video.py --base configs/cogvideox_5b.yaml configs/inference.yaml --seed $RANDOM"
+run_cmd="$environs python sample_video.py --base configs/cogvideox_2b.yaml configs/inference.yaml --seed $RANDOM"

Next I had a module import error - it seems that triton is required, but either I have missed it or it is not listed in requirements.txt? pip install triton fixed that, but I then had to manually patch out the import and arg parsing for deepspeed since I was not yet able to install DeepSpeed on my system.
After that I was able to run the inference script, which only showed activity on GPU 0, the two remaining GPUs stayed idle. As a result I ran into an OOM:
[rank0]: torch.cuda.OutOfMemoryError: HIP out of memory. Tried to allocate 35.31 GiB. GPU

Is my understanding correct, that the script was supposed to spread the model across all available GPUs and thus utilize the memory of all three cards?
PyTorch does see all of my cards, so the overall setup should be OK:
>>> import torch
>>> for i in range(torch.cuda.device_count()):
...    print(torch.cuda.get_device_properties(i).name)
...
AMD Radeon Instinct MI25
AMD Radeon Instinct MI25
AMD Radeon Instinct MI25

Do you have any ideas on how to figure out why it is not utilizing all three cards? Is there anything in the code I could try to modify or perhaps print some debugging info which would help us to find the problem?

哦，这个问题似乎是因为在 device_map 的问题，在sat版本，我们都载入到一张卡，而在diffusers的实现，设定divice_map = balanced 能均匀的分到三张卡

The `device_map = "balanced" was a good hint, I totally missed that! So retried the diffusers version, enabling the balanced option now indeed shows some activity on all three GPUs, but overall - no luck :(
torch.cuda.OutOfMemoryError: HIP out of memory. Tried to allocate 35.31 GiB. GPU  has a total capacity of 15.98 GiB of which 10.98 GiB is free.

It is still not clear to me why it is trying to allocate 35GB, which is way more than listed in the table we talked about earlier? I found these issues though:
https://github.com/pytorch/pytorch/issues/112997
https://github.com/THUDM/CogVideo/issues/92
Sounds like I ran into the same problem?

I don't think it's because of the balanced reason. I want to know if only one GPU was running the whole time (until the OOM error occurred), rather than multiple GPUs, or if the OOM was encountered during inference after the model had already been successfully loaded

I think it happens during inference, but I will rather paste the full log, I think you will have a better idea when you see it:
Loading pipeline components...:   0%|                     | 0/5 [00:00<?, ?it/s]
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
Loading checkpoint shards: 100%|██████████████████| 2/2 [00:46<00:00, 23.21s/it]
Loading pipeline components...: 100%|█████████████| 5/5 [01:03<00:00, 12.78s/it]
  0%|                                                    | 0/50 [00:00<?, ?it/s]/home/user/.local/lib/python3.12/site-packages/diffusers/models/attention_processor.py:1925: UserWarning: 1Torch was not compiled with memory efficient attention. (Triggered internally at ../aten/src/ATen/native/transformers/hip/sdp_utils.cpp:505.)
  hidden_states = F.scaled_dot_product_attention(
  0%|                                                    | 0/50 [00:00<?, ?it/s]
Traceback (most recent call last):
  File "/home/user/Work/CogVideo/inference/cli_demo.py", line 107, in <module>
    generate_video(
  File "/home/user/Work/CogVideo/inference/cli_demo.py", line 67, in generate_video
    video = pipe(
            ^^^^^
  File "/home/user/.local/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/.local/lib/python3.12/site-packages/diffusers/pipelines/cogvideo/pipeline_cogvideox.py", line 687, in __call__
    noise_pred = self.transformer(
                 ^^^^^^^^^^^^^^^^^
  File "/home/user/.local/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/.local/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/.local/lib/python3.12/site-packages/accelerate/hooks.py", line 166, in new_forward
    output = module._old_forward(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/.local/lib/python3.12/site-packages/diffusers/models/transformers/cogvideox_transformer_3d.py", line 458, in forward
    hidden_states, encoder_hidden_states = block(
                                           ^^^^^^
  File "/home/user/.local/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/.local/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/.local/lib/python3.12/site-packages/accelerate/hooks.py", line 166, in new_forward
    output = module._old_forward(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/.local/lib/python3.12/site-packages/diffusers/models/transformers/cogvideox_transformer_3d.py", line 131, in forward
    attn_hidden_states, attn_encoder_hidden_states = self.attn1(
                                                     ^^^^^^^^^^^
  File "/home/user/.local/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/.local/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/.local/lib/python3.12/site-packages/accelerate/hooks.py", line 166, in new_forward
    output = module._old_forward(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/.local/lib/python3.12/site-packages/diffusers/models/attention_processor.py", line 490, in forward
    return self.processor(
           ^^^^^^^^^^^^^^^
  File "/home/user/.local/lib/python3.12/site-packages/diffusers/models/attention_processor.py", line 1925, in __call__
    hidden_states = F.scaled_dot_product_attention(
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
torch.cuda.OutOfMemoryError: HIP out of memory. Tried to allocate 35.31 GiB. GPU  has a total capacity of 15.98 GiB of which 10.98 GiB is free. Of the allocated memory 4.29 GiB is allocated by PyTorch, and 472.07 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_HIP_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)

The state of the GPUs when the OOM happened was as follows:

My naive interpretation was, that it may have to do with attention after all, the backtrace where it OOMs points to hidden_states = F.scaled_dot_product_attention( and PyTorch issues a warning in the beginning, saying that:

attention_processor.py:1925: UserWarning: 1Torch was not compiled with memory efficient attention. (Triggered internally at ../aten/src/ATen/native/transformers/hip/sdp_utils.cpp:505.)

This is why I thought it may be related to https://github.com/pytorch/pytorch/issues/112997
Just to be sure I did not mess anything up, here is the patch of my changes for using the 2B model with multiple GPUs:
diff --git a/inference/cli_demo.py b/inference/cli_demo.py
index 323e9af..85588a6 100644
--- a/inference/cli_demo.py
+++ b/inference/cli_demo.py
@@ -43,18 +43,19 @@ def generate_video(
     # add device_map="balanced" in the from_pretrained function and remove the enable_model_cpu_offload()
     # function to use Multi GPUs.

-    pipe = CogVideoXPipeline.from_pretrained(model_path, torch_dtype=dtype)
+    pipe = CogVideoXPipeline.from_pretrained(model_path, torch_dtype=dtype, device_map="balanced")

     # 2. Set Scheduler.
     # Can be changed to `CogVideoXDPMScheduler` or `CogVideoXDDIMScheduler`.
     # We recommend using `CogVideoXDDIMScheduler` for CogVideoX-2B and `CogVideoXDPMScheduler` for CogVideoX-5B.
     # pipe.scheduler = CogVideoXDDIMScheduler.from_config(pipe.scheduler.config, timestep_spacing="trailing")
-    pipe.scheduler = CogVideoXDPMScheduler.from_config(pipe.scheduler.config, timestep_spacing="trailing")
+    #pipe.scheduler = CogVideoXDPMScheduler.from_config(pipe.scheduler.config, timestep_spacing="trailing")
+    pipe.scheduler = CogVideoXDDIMScheduler.from_config(pipe.scheduler.config, timestep_spacing="trailing")

     # 3. Enable CPU offload for the model, enable tiling.
     # turn off if you have multiple GPUs or enough GPU memory(such as H100) and it will cost less time in inference
-    pipe.enable_model_cpu_offload()
-    pipe.enable_sequential_cpu_offload()
+    #pipe.enable_model_cpu_offload()
+    #pipe.enable_sequential_cpu_offload()
     pipe.vae.enable_slicing()
     pipe.vae.enable_tiling()

My command line is:
python cli_demo.py --model_path=/home/user/Work/models/CogVideoX-2b --dtype=float16 --prompt "a fallow doe walking on mars"
Please let me know if there is anything I can debug to help narrow down the issue.

This code is completely correct, but it seems that it cannot run on multiple AMDGPUs
pipe.enable_model_cpu_offload()
pipe.enable_sequential_cpu_offload()
Can these two sentences enable you to run on an single AMD GPU?
Regarding the multi-card issue you mentioned, perhaps it should indeed be addressed by the diffusers or PyTorch team. It cannot be handled at the moment because your model is in the normal loading stage (around 15GB), which is normal.

pipe.enable_model_cpu_offload()
pipe.enable_sequential_cpu_offload()
Can these two sentences enable you to run on an single AMD GPU?

Unfortunately no, when I remove the "balanced" setting and let the above two lines in, then something keeps loading and loading until I run out of RAM (I have only 64GB), then the system starts swapping and becomes unresponsive. The GPUs stay idle and do not show any activity.

Regarding the multi-card issue you mentioned, perhaps it should indeed be addressed by
the diffusers or PyTorch team. It cannot be handled at the moment because your model
is in the normal loading stage (around 15GB), which is normal.

I am somewhat confused though, what does the table mean by "single GPU RAM consumption: 3.6GB"? Is this meant as "additional VRAM needed on top of the loaded model"?
So, to make sure I understand the issue correctly: I need to be able to load the whole model onto one GPU, this can not be split across multiple GPUs? And then inference could be split between multiple GPUs, but I never get that far, because I can't load the model? Is my understanding of the problem correct?

No, the 3.3GB refers to the data obtained from testing on NVIDIA GPUs after using pipe.enable_sequential_cpu_offload(). In the data explanation, we also mentioned that the tests were conducted on A100/H100 GPUs, and these GPUs can function properly with this command on a single card. Unfortunately, in your test, AMDGPU did not work properly.
When pipe.enable_sequential_cpu_offload() is enabled, the model is not fully loaded onto the GPU, which allows for a significant reduction in VRAM usage by interacting between the GPU and CPU.

Aha... wait, so this means that using multiple GPUs and enabling CPU offload are mutually exclusive? Is there any potential way to have CPU offload together with the use of all available GPUs?
My suspicion is that pipe.enable_sequential_cpu_offload() generally did work for me, but the 64GB of system RAM were not enough and my system swapped itself to death before I could see any GPU action. I wonder you see a way, that everything could be spread out between system RAM while still using the "balanced" option? In the meantime I'll see if I can order another 64GB for the system and test with enabled CPU offload again.

Yes, it is mutually exclusive, only one can be chosen, 64GB of free RAM is sufficient, the issue does not lie here

For multi-GPU cards, due to the design in the diffuser library, pipe.enable_sequential_cpu_offload() must be disabled. This is not something that can be determined by the model layer and requires adherence to the design in the diffuser library.

guys is there a way to use selected gpu only? Config etc?

guys is there a way to use selected gpu only? Config etc?

I think you can simply control the visibility of your GPUs using the CUDA_VISIBLE_DEVICES environment variable to point CogVideo to the specific GPU which you want to use. I think this should work.

guys is there a way to use selected gpu only? Config etc?

I think you can simply control the visibility of your GPUs using the CUDA_VISIBLE_DEVICES environment variable to point CogVideo to the specific GPU which you want to use. I think this should work.

thank you, will try


cleaning history section is not integrated with dark mode

texts not seen
http://prntscr.com/jpyu5d

Looks bad, but you are not reaching the right people through this issue tracker. I'll see if I can find someone to tell but better is if you report it at https://bugs.opera.com/wizard/ .


Handle more items in inventory

Currently with more items we get an ugly scroll bar. Solutions:

set a maximum number of items
make the scroll bar pretty
redesign the inventory mode and show items at a side

GOAT should be able to handle more items for the future games, especially escape rooms. They can still be displayed on the side, but icons should be added dynamically, with 0 for an empty inventory. If the list grows too big, it should be wrapped (using one of Godot's addons with improved container features). That way all items can be displayed at once. Drag and drop will work the same way as it does now.
The change has to be applied to the InventoryBar scene as well.
Unfortunately, this has to include changes in the context menu, otherwise the game will still be effectively restricted to 8 items. So, either a temporary fix is used until the "hold item in hand" feature is implemented, or this task will be done after "hold item in hand".


Sharing Schema / Models

Problem
We have a few patterns we'd like to share across projects that use prisma, for example, a message outbox pattern, where the tables would ideally be consistent across projects and we'd like to distribute the table definitions with the source code.
Suggested solution
Allow some form of import / additional models file(s) for client generation in the schema syntax:
datasource db {
  provider = "postgresql"
  url      = env("MIGRATION_POSTGRES_URI")
}

// ----- Models -----
model Customer {
  ...
}

include OutboxLib {
  models =<EMAIL_ADDRESS>}

Or perhaps a syntax as follows could allow for eventual overrides / customization:
// Allows end-user to customize / stamp out tables
model CustomerTopicOutboxMessages {
  // file path reference, name of the model we're pulling out
<EMAIL_ADDRESS>OutboxMessages)
}

model InvoiceTopicOutboxMessages {
<EMAIL_ADDRESS>OutboxMessages)
}

Alternatives
We looked a little bit at the client extensions, but it didn't seem like that would integrate with the migration product.
We could just ask devs to copy/paste, but this feels like a quick way to ensure these "standard" tables become anything but.

Closing as duplicate of https://github.com/prisma/prisma/issues/2377
I can't say much about this beside that we want to do something "in the future", hopefully "soon".


ARIA-277 Support for Type Qualified Name

Can one of the admins verify this patch?

Could you please check if all my commits are squashed now ? Am bit new to this squash concept.

Yeah, no problem. Basically as long as github says this PR has 9 commits, the commits were not fully squashed. you can find a quick answer to how to squash here, but please be careful, i would copy the working dir for the forked repo and try squashing from that copy (only when you are 100% sure all of your changes are present in the single commit, you should push back to your branch).

Hi Max,
Finally i was able to squash all the commits and resolve the merge issue with a single commit. Kindly check and let me know if its fine this time :)

looks good :smiley: will merge into master shortly

This PR was merged to master, can you please erase the PR?

Thanks max. I hope erasing the PR means closing this PR ?


But might want to consider moving the translation away from the ui.


Enter animation for sidebar

It would be really neat if there could be an entering animation for the sidebar like this one:

How would you envision the animations going?
The animation on the Timeline itself are always traveling up or down, based on the direction in the timeline that you're going. For the watchface, you're in the present, so we can't really say up or down. What do you think?

I meant for only entering animation, so when the watchface is loaded the taskbar will make an animation from bottom to top kind of like the weather icon in the gif

Especially now that Pebble has change the watchface entry animation so that it just slides in, I don't think this is needed any longer


Bertha can display two IP addresses for the same adapter (one is old/…

…wrong)

Hi @rachitb777, I'm your friendly neighborhood Microsoft Pull Request Bot (You can call me MSBOT). Thanks for your contribution!
You've already signed the contribution license agreement. Thanks!
The agreement was validated by Microsoft and real humans are currently evaluating your PR.
TTYL, MSBOT;


Add irqbalance

irqbalance distributes hardware interrupts across processors. If there is only 1 CPU it will silently exit.
Increases performance in multiprocessor/multicore systems.
Seems to reduce "CPU Stuck" issue with nested virtualization.

@amiga23 thanks for this contribution. I've merged thus but I do not believe that it will be started automatically without further changes.see https://github.com/rancher/k3os/blob/e4055f23265b65826726331f6e7c58e6ed65133e/overlay/libexec/k3os/boot#L35-L68


Missing pairs

Minor fixes and pre-tests run on directories with missing pairs, both with and without sampInfo input

i'm testing this pr and got an error when rendering the r markdown:
processing file: Descriptive_Statistics.rmd
  |...                                                                   |   5%
   inline R code fragments

  |......                                                                |   9%
label: setup (with options)
List of 1
 $ include: logi FALSE

  |..........                                                            |  14%
  ordinary text without R code

  |.............                                                         |  18%
label: load and parse (with options)
List of 1
 $ include: logi FALSE

  |................                                                      |  23%
  ordinary text without R code

  |...................                                                   |  27%
label: overall dist (with options)
List of 1
 $ fig.height: num 4

  |......................                                                |  32%
   inline R code fragments

  |.........................                                             |  36%
label: class with facet (with options)
List of 1
 $ eval: logi TRUE

Quitting from lines 101-129 (Descriptive_Statistics.rmd)
Error in labs(y = "Num of WSI", caption = Chi) : object 'Chi' not found
Calls: <Anonymous> ... eval_with_user_handlers -> eval -> eval -> labs -> dots_list
In addition: Warning message:
Removed 965 rows containing non-finite values (stat_bin).

Execution halted

New push runs beginning to end locally (align and the stats.rmd) using the mis-paired sample files.

i am getting a different error now. i included the tcga ids at the bottom of this comment.

processing file: Descriptive_Statistics.rmd
  |...                                                                   |   5%
   inline R code fragments

  |......                                                                |   9%
label: setup (with options)
List of 1
 $ include: logi FALSE

  |..........                                                            |  14%
  ordinary text without R code

  |.............                                                         |  18%
label: load and parse (with options)
List of 1
 $ include: logi FALSE

  |................                                                      |  23%
  ordinary text without R code

  |...................                                                   |  27%
label: overall dist (with options)
List of 1
 $ fig.height: num 4

  |......................                                                |  32%
   inline R code fragments

  |.........................                                             |  36%
label: class with facet (with options)
List of 1
 $ eval: logi FALSE

  |.............................                                         |  41%
  ordinary text without R code

  |................................                                      |  45%
label: continuous with facet (with options)
List of 1
 $ eval: logi FALSE

  |...................................                                   |  50%
  ordinary text without R code

  |......................................                                |  55%
label: km univar (with options)
List of 2
 $ fig.height: num 6
 $ eval      : logi TRUE

  |.........................................                             |  59%
  ordinary text without R code

  |.............................................                         |  64%
label: km facet (with options)
List of 3
 $ fig.height: num 8
 $ fig.width : num 12
 $ eval      : logi FALSE

  |................................................                      |  68%
  ordinary text without R code

  |...................................................                   |  73%
label: cox cat univar (with options)
List of 3
 $ fig.height: num 3
 $ fig.width : num 8
 $ eval      : logi TRUE

Quitting from lines 220-232 (Descriptive_Statistics.rmd)
Error in axisTicks(rangeb/2, log = TRUE, nint = 7) :
  log - axis(), 'at' creation, _LARGE_ range: invalid {xy}axp or par; nint=7
	 axp[0:1]=(1e-307,1e+308), usr[0:1]=(0,inf); i=615, ni=87
Calls: <Anonymous> ... eval_with_user_handlers -> eval -> eval -> ggforest -> axisTicks
In addition: Warning messages:
1: In eval(expr, envir, enclos) :
  No additional sample information provided beyond survival and TIL. Only univariable analyses will be performed
2: Removed 965 rows containing non-finite values (stat_bin).
3: In coxph.fit(X, Y, istrat, offset, init, control, weights = weights,  :
  Loglik converged before variable  1 ; coefficient may be infinite.

Execution halted

here are the tcga ids i am using in the test
.
├── til
│   ├── prediction-TCGA-3C-AALI-01Z-00-DX1
│   ├── prediction-TCGA-3C-AALJ-01Z-00-DX1
│   ├── prediction-TCGA-3C-AALK-01Z-00-DX1
│   ├── prediction-TCGA-4H-AAAK-01Z-00-DX1
│   ├── prediction-TCGA-5L-AAT0-01Z-00-DX1
│   ├── prediction-TCGA-5L-AAT1-01Z-00-DX1
│   ├── prediction-TCGA-5T-A9QA-01Z-00-DX1
│   ├── prediction-TCGA-A1-A0SB-01Z-00-DX1
│   ├── prediction-TCGA-A1-A0SD-01Z-00-DX1
│   ├── prediction-TCGA-A1-A0SE-01Z-00-DX1
│   ├── prediction-TCGA-A1-A0SF-01Z-00-DX1
│   ├── prediction-TCGA-A1-A0SH-01Z-00-DX1
│   ├── prediction-TCGA-A1-A0SI-01Z-00-DX1
│   ├── prediction-TCGA-A1-A0SJ-01Z-00-DX1
│   ├── prediction-TCGA-A1-A0SK-01Z-00-DX1
│   ├── prediction-TCGA-A1-A0SM-01Z-00-DX1
│   ├── prediction-TCGA-A1-A0SN-01Z-00-DX1
│   ├── prediction-TCGA-A1-A0SP-01Z-00-DX1
│   ├── prediction-TCGA-A1-A0SQ-01Z-00-DX1
│   └── prediction-TCGA-A2-A04N-01Z-00-DX1
└── tumor
    ├── prediction-TCGA-3C-AALI-01Z-00-DX1
    ├── prediction-TCGA-3C-AALI-01Z-00-DX2
    ├── prediction-TCGA-3C-AALJ-01Z-00-DX1
    ├── prediction-TCGA-3C-AALJ-01Z-00-DX2
    ├── prediction-TCGA-3C-AALK-01Z-00-DX1
    ├── prediction-TCGA-4H-AAAK-01Z-00-DX1
    ├── prediction-TCGA-5T-A9QA-01Z-00-DX1
    ├── prediction-TCGA-A1-A0SB-01Z-00-DX1
    ├── prediction-TCGA-A1-A0SD-01Z-00-DX1
    ├── prediction-TCGA-A1-A0SE-01Z-00-DX1
    ├── prediction-TCGA-A1-A0SF-01Z-00-DX1
    ├── prediction-TCGA-A1-A0SH-01Z-00-DX1
    ├── prediction-TCGA-A1-A0SI-01Z-00-DX1
    ├── prediction-TCGA-A1-A0SJ-01Z-00-DX1
    ├── prediction-TCGA-A1-A0SK-01Z-00-DX1
    ├── prediction-TCGA-A1-A0SM-01Z-00-DX1
    ├── prediction-TCGA-A1-A0SN-01Z-00-DX1
    ├── prediction-TCGA-A1-A0SP-01Z-00-DX1
    ├── prediction-TCGA-A1-A0SQ-01Z-00-DX1
    └── prediction-TCGA-A2-A04N-01Z-00-DX1


Tutorials section appears under 'Documentation', based on top navigaton indication

This is just a note! I know it is all a draft, but just mentioning it while I see it, because backing out of Tutorials leads to 'Tools', so the navigation in and out respectively implies different (confusing) nested organizational frameworks.

IIUC we've discussed this since this issue was raised, and is no longer a problem. @juliebates please reopen if I've missed something.

Backing out of Tutorials no longer leads to Tools for me either.
I do notice some other things which may or not be relevant:
-The Tutorials button on the main page is not active, it points to a 404 error page
-Getting Started with ReproNim button goes forward to Tutorials via a Left margin Navigation list that includes Tutorials
-From within the Tutorials main window, (which is currently a brief list of How to....), the top horizontal 'directory tree' shows Resources > Tutorials.  (thiat isn't how I got there, but that may be fine, it may be how you want user to orient)
-but also note that if I hit the back arrow from that preliminary list window, it takes me to the actual DICOM to BIDS tutorial)
--and then if I back out another step from the full DICOM to BIDS tutorial, it takes me to the 'Getting Started with ReproNim' window, which by the way shows as beign a subsidiary of Resources.
I'm not sure which if any of those things are helpful, i"m just relaying my observations in the moment, but I also realize that things are actively in development and likely to move around


Note about aws cli Default output format as json

Documentation update to address #110

If you've the time, a PR for the code change suggested on the issue would be better.


Still maintaining?

Hi,
are you still maintaining it? If not, can we move it into my namespace? I have made several improvements and racing ahead..
Cheers,
Stefan

Hi Stefan,
I'm OK to move it  to your namespace. Please tell me how to proceed.
Cheers,
BG

Hi, can you please transfer it to the FOSSi Foundation organization? Just go to settings->options-> Transfer ownership and then Transfer. As target use "fossi-foundation". I want to maintain it there. Thanks!

https://help.github.com/en/articles/transferring-a-repository#transferring-a-repository-owned-by-your-personal-account


Binary indices occasionally corrupt

(continuation from #47)
I have users where the lokijs binary indices occasionally corrupt. It's hard to find out how this happens -- it happens infrequently, and a long-running log to catch is would generate very large files for all users. I can't reproduce this at will. Right now I'd be happy to see 1.3.5 drop which has detect and repair functions.

Great I will work over the next few days/weekend to get things prepared for a 1.5.3 release.  When that is done I will see if @techfort can publish an out-of-band release.
We can continue discussions during diagnosis here.

I have clone on for all collections, so it's not an object being modified without an update being called.

Yea weird... so what is the datatype of the property whos binary index is becoming corrupt?   Do I recall that you convert DateTimes from our stored numbers back to a date data type?  If so maybe we should focus there.
You also might look at disabling the adaptiveIndices so that they revert to our original 'lazy' indices.  So whenever we insert/update/remove we would flag index as dirty and it would stay dirty until an indexed find occurred on that index, at which point we would do a full (expensive) rebuild of index which will stay good until the next time they insert/update/remove.  That is controlled by adaptiveBinaryIndices: false constructor option or passing to collection.configureOptions().

The datatype of the property is 'number' (more specifically, an integer).
I did convert DateTimes from strings to JS Dates, but that was for a field in the doc, not the LokiJS metadata, and I've since removed that field since with cloning on, the results from searches weren't re-inflated, so I got strings where I expected Date objects.
How do I disable adaptiveIndices for existing DBs? By calling collection.configureOptions() on the loaded collection? Or can I do this during load in the store object?

Yes you can call configureOptions after db has loaded to switch back and forth.
That method currently only supports that one option but it might be expanded in the future to allow switching other config options.

Diagnostics are going to be hard in my case, as I can't trigger it, it doesn't occur often (only two known cases in all the time I've used LokiJS), and when I do get to hear about it, there is no history. I have no idea how long I'd have to wait for nonoccurence of the problem before I could decide that disabling adaptiveBinaryIndices would have any effect.
A 10 minute test with logging will easily produce several megabytes of logging; the host application also generates debug logging, so I can't easily whittle this down. To have logging running by default would consume a lot of memory (the host application collects the log in RAM) and disk space (where I'd have to accumulate the logs) over the course of even just a week, so I really can't just enable this for everyone in the hopes of catching the problem as it happens.

Right, and disabling adaptive binary indices may just hide the problem since we forcibly rebuild the index all the time but it the performance is adequate, it might calm things down your main branch to use while you test out diagnostic builds possibly on others (or behind options/flags).
So one thought is that we enable loki changes api along with timer (in your code)... we might set the timer to 1 minute and do full index validation and if it is valid we clear changes and wait for next timer.  Only if it is invalid might we save the changes to a file for subsequent diagnosis.  That would definitely run slower but for a diagnostic build that could be expected.

The typical DB won't be so big to make this a problem I'd guess. 30k items is considered very big, 500k is actively recommended against for the host application with or without my plugin, and most people will have only about 1k-5k docs in their DB.
Diagnostic builds can be as slow as they need to be; the multi-megabyte logs certainly slow things down quite a bit. But the problem is people only notice the problem when the corruption has already happened on a non-diagnostic build. I'm OK with putting out production builds that do background index validation (I have existing infrastructure to trigger activity when the host app is idle), but that would get us the state of things soon after it happened, but not the lead-up.

(also, detect-and-fix is sufficient workaround for my use-case as the problem will most likely manifest itself to the user during startup. If I can detect-and-fix during startup, I'm likely good to go from a pragmatic perspective).

Oh yeah this is perhaps relevant -- I haven't seen complaints that point to this problem during runtime of the app, just in the access I do directly after loading the DB. This could well be however because users have gotten into the habit of restarting when they experience problems, and the index corruption would prevent successful startup, forcing them to seek support.

I might very well be to initial and/or final load/save.
So the life of your plugin / session is (roughly) the same as the life of the browser instance?  This is an add in that loads with browser and is shutdown with browser?  What about multiple browser instances running separate copies, saving to the same database?  That might cause problems if two loki instances are competing with the same database.  I would assume multiple tabs within the same browser would use the same loki instance but multiple browsers would run multiple loki instances?
I have been working to get typescript definitions updated with 1.5.3 changes and that looks like it should be done soon, so I can go ahead and ask for a publish of 1.5.3.

The lifetime of my plugin is the same as the run time of the Zotero instance. Zotero uses firefox as a base but isn't really a browser, and only one can be running at any time. Even when it still lived entirely in FF, Zotero and my plugin run in what's known as Firefox "chrome mode" javascript (not to be confused with running in the Chrome browser) which means it runs in the global browser scope, not any page scope, wholly separate from what tabs may or may not be doing.
The lokijs database in question is also loaded from and saved to sqlite transactionally, so I currently see no route where the save could corrupt the index -- the collection goes to disk, indexes and all, or it doesn't. Even if there were multiple copies of Zotero running, Zotero locks the sqlite database on startup, so it's not even possible to change the DB from the command line while Zotero is running.

Oh, good... one less thing to worry about.   So having settled that, if multiple instances of loki pointing to -different- database names should be perfectly acceptable.  I wonder if it might be useful to set up a 'metadata' / diagnostic database where we might store lightweight summary information.  On shutdown we might store some kind of record of shutdown and possibly all collection counts at the time of shutdown, and possibly index lengths.  We could do the same on startup.  I am just thinking about how we might start adding structured diagnostic info to maintain control over diagnostics.  Let me know if that would add additional complexities as i am not sure how that meta db could be submitted to you for diagnostics.

I'd love to see that. I already have a facility to deliver debug data to me, adding a new section to the submission package is trivial.

Looks like @techfort has already published 1.5.3. Feel free to go ahead and try it out :smile:

Alright, I've put out a new release with 1.5.3.

I have a user that has this error:
this.data[biv[1]] is undefined
["../node_modules/lokijs/src/lokijs.js"]/</</Collection.prototype.checkIndex
["../node_modules/lokijs/src/lokijs.js"]/</</Collection.prototype.checkAllIndexes

I'm getting around this uwing the following for now:
try {
      coll.checkAllIndexes({ repair: true })
    } catch (err) {
      coll.ensureAllIndexes(true)
    }

I have the DB that triggers it, but as before, not the lead-up.

There may have been only one document / one index value in the array and I think I wasn't handling that correctly.  I have checked in a fix to that.
So you have a copy of the database which contains this corruption?  If so, you might loading into LokiSandbox and then run the autocreated script which dumps contents to log and inspector.  Browse the database with the object inspector and see if you can find anything out of the ordinary.  You can also modify the script to query against or run the checkIndex methods to recreate the error and hopefully narrow down the collection containing the index which is invalid.  So long as you don't redefine the 'db' object it will continue to exist across script runs.

I do have a copy of the database, and the collection that triggered the error does indeed have one record.

Ok well a collection with one record will always throw an exception when calling checkIndex so your checkAllIndexes will throw exception if any one of the indexes has only one value (this should be fixed in the next release with the above commit).
Until the next release, you might try iterating collections and calling checkIndex on each of them... either with a try catch or not calling at all if collection.count() < 2.

This would also work correct?
try {
  coll.checkAllIndexes({ repair: true })
} catch (err) {
  coll.ensureAllIndexes(true)
}

As I mentioned if any one of the collections has one document you will be rebuilding all indexes.  It would be safe to do the above but possibly excessively time consuming.
I would do something like :
db.collections.forEach(functioni(coll) {
  if (coll.count !== 1) {
    coll.checkIndex({ repair: true});
  }
});

I don't understand. My code calls ensureAllIndexes on the specific collection that fails -- how does that affect other collections?

Ah your right that should work... was thinking of need to check all collections but we don't have a db level method.


Patch CVE-2023-27533 in cmake

Merge Checklist

All boxes should be checked before merging the PR (just tick any boxes which don't apply to this PR)

[x] The toolchain has been rebuilt successfully (or no changes were made to it)
[x] The toolchain/worker package manifests are up-to-date
[x] Any updated packages successfully build (or no packages were changed)
[x] Packages depending on static components modified in this PR (Golang, *-static subpackages, etc.) have had their Release tag incremented.
[x] Package tests (%check section) have been verified with RUN_CHECK=y for existing SPEC files, or added to new SPEC files
[x] All package sources are available
[x] cgmanifest files are up-to-date and sorted (./cgmanifest.json, ./toolkit/scripts/toolchain/cgmanifest.json, .github/workflows/cgmanifest.json)
[x] LICENSE-MAP files are up-to-date (./LICENSES-AND-NOTICES/SPECS/data/licenses.json, ./LICENSES-AND-NOTICES/SPECS/LICENSES-MAP.md, ./LICENSES-AND-NOTICES/SPECS/LICENSE-EXCEPTIONS.PHOTON)
[x] All source files have up-to-date hashes in the *.signatures.json files
[x] sudo make go-tidy-all and sudo make go-test-coverage pass
[x] Documentation has been updated to match any changes to the build system
[ ] Ready to merge

Summary

What does the PR accomplish, why was it needed?
Patch CVE-2023-27533 in cmake.
Change Log

CVE-2023-27533

Does this affect the toolchain?

YES
Associated issues

NA

Links to CVEs

https://nvd.nist.gov/vuln/detail/CVE-2023-27533

Test Methodology

Pipeline build id: xxxx

Thank you @PawelWMS and @mbykhovtsev-ms for the reviews.

Closing this PR as these changes are taken into the PR #10484


Pass the request, response and next() arguments to the middlewares

Hello,
Is it possible that the re, res, next() arguments can be passed directly to the controllers as arguments? To streamline it with the approach taken by express. Currently the first argument is the context while the second is the next() when available.

Hey @imolorhe.
Rill, much like koa, is designed with the future of javascript in mind. The reason why everything is bundled under context is to keep the function signature for middleware simple.
The best way to use the context is to pull out properties using es2015 argument destructuring. Like so:
app.get(({ req, res, locals }, next) => ...)

The other big benefits here are that there can be much more than 'req' and 'res' (notice locals) making the context more flexible without polluting those existing objects.
I think this is best for the library to keep the signature as is but I certainly welcome all feedback and if you have any other ideas or questions please bring them forward 😀.


Deoplete sources support

Thanks for create this plugin for php, I want add these to Spacevim, but I can not find deoplete source , even though deoplete support omni sources, but it will be better, if phpcd has a deoplete sources.

Please read :h g:deoplete#omni_patterns (or even :h deoplete.txt) first. :smile:
If you really know what are you doing, you could put the following lines into you vimrc file:
let g:deoplete#omni_patterns = {}
let g:deoplete#omni_patterns.php = '->\|::'

As both deoplete and phpcd.vim support the omnifunc, there is nothing to do to make deoplete support phpcd.vim.
If you have an Plug named XXX supporting the completion of language XXX, what you need to do is just set the deoplete#omni_patterns.xxx variable to let deoplete to auto trigger the omnifunc. And if you do not have deoplete, the C-xC-o is your friend.

I know the omni sources works well, and I think you can have a look at javacomplete 2,  there is a issue about why we add deoplete source instead of using omni.
And if you think it is uesful, please reopen this issue.

Omni is not really async ,use javacomplete2 source make deoplete more fast.

@lvht you can have a look at my fork, I will push the deoplete source.

@wsdjeg sorry for misunderstanding. But I am not familiar with deoplete. PR is welcomed 😄

Thanks, I will try to do it.


Minor changes and syntax improvements

Changed Optional[bool] to bool for only_production to restrict the possible values from True, False, None to just True and False.
Changed dict(json=json.dumps(...)) to {"json": json.dumps(...)} as per Pylint(R1735:use-dict-literal)
Changed if prodfilter: to if prodfilter is not None: as per PEP-8
longer variable names, more type hints, f-strings, black formatting

Hey there! Thanks for submitting this PR. I really appreciate it, and these are all great ideas. I haven't added black to the workflow checks yet either so it is certainly nice to have better formatting.
It looks however like the PR is failing both checks (pytest is giving an authorization error for the API) and the linting (ruff, isort). Can you confirm if these tests are working when you run them?

Ran isort again, should be good now. Not sure about the tests though and where it would get the API tokens from.

Awesome, thanks! The end-to-end test requires the action secrets SIDEFX_CLIENT and SIDEFX_SECRET (You can get credentials here). Though looking at the diff I don't think the behaviour will change anyway. Approved :)


[CV request] We need a cv param for mzQC as a file (just like MS:1000584)

Purpose: The term is to be used to specify the file type of mzQC. E.g. as tag for source/input files, also in other formats as mzML.
Name: mzQC format
Definition: Proteomics Standards Initiative mzQC format for quality control data. is_a: MS:1001459 ! file format
Comment: None
Value type: N/A
Unit: N/A
Category: Structural term.

I would advocate to rather put this into the PSI-MS-CV, together with the "software generating QC metrics", which are still a stub at the very beginning of our current obo.
The reasoning for this is, that the file format should also be known by a consuming software, which only reads the PSI-MS-CV and might not be interested into QC.

or maybe is_a: MS:1001040 ! intermediate analysis format ?

see https://github.com/HUPO-PSI/psi-ms-CV/pull/71

Need it here #131

I think the request in the MS CV is better than having it as an intermediate format. 👍


Issue 57130 Fix GLTFDocument.generate_scene if state is null

This should fix #57130 . I just added a check for null state, if we need more sophisticated error handling, please let me know ;)

The fix seems good. Just a nitpick on the commit title for future PRs, we prefer the commit title to only refer to what the commit does, and mention the issue (with the # that will properly generate a backreference) in the body of the commit message, i.e.:
Fix `GLTFDocument.generate_scene` crash with invalid state

Fixes #57130.

Thanks! I'll keep it in mind for future PRs :)

Thanks!


More reasonable and restrained cuts to Section 3 of the introduction

This section is pretty powerful so I didn't want to make any big changes that could hurt that. But in a few places I thought it could be shortened.
One big question I have about this section is the last paragraph. I think this paragraph is what my kind of dopey trailer in #55 was trying to be. What do you think of moving this section ("The browser and me") before the previous one ("Explaining the black box")? And then using the last paragraph of this section as the last paragraph of "Explaining the black box"?

You're right, the "and" works well. I'll let you merge this if you like it.


Framework to handle duplicate identifiers via postprocessing

Not ready to merge due to https://github.com/chamons/Embeddinator-4000/commit/6e4fb6403fe30833120f6edfcc65cc133f7bfc6e but I wanted to get something out there.

Build and tests passed.

I fixed the mentioned test case with this extra hack:
https://github.com/mono/Embeddinator-4000/pull/156/commits/9c5de99788988bd8806d32e8d4971ef2c2df7553
Ideally we should be moving such logic out of GetSignatures but since we're not passing in the right types, it'd involve adding yet another boolean in, which does not seem better.

Build and tests passed.

Ok, I thought it was fixed but it isn't...

Better add them in Tests.m so building tests will fail if the logic breaks :)

Build and tests passed.

Merge integration broke a test. Fixing

Build and tests passed.


Update index.md

Some minor text edits.

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.:x: JohnA-Oslo sign nowYou have signed the CLA already but the status is still pending? Let us recheck it.

@JohnA-Oslo John - Thank you for the edit. I tweaked just to align to Microsoft doc conventions. We appreciate the help to improve our guidance -- Monica


Feature Request - ECommerce Tracking

Hey, first of all: Thanks for this awesome package :)
I think it would be great to have ECommerce Tracking Functionality.

I opened a pull request for Ecommerce tracking: https://github.com/poitch/dart-matomo/pull/27

Merged and published. Thank you for contributing.