exception after installation (undefined method `filter')

Rails 7.1.3
Ruby 3.3.0
I added the gem to the development and test section of my Gemfile.  Added the route and the debugbar_javascript to my application layout.
However, I'm getting the following error:
$ rails test
[...]
.rvm/gems/ruby-3.3.0/bundler/gems/debugbar-eb2430db8435/lib/debugbar/subscribers/active_record.rb:38:in `block in sql': undefined method `filter' for class Debugbar::ActiveRecordEventSubscriber (NoMethodError)

            filtered_params = filter(attribute_name, casted_params[i])
                              ^^^^^^
	from .rvm/gems/ruby-3.3.0/bundler/gems/debugbar-eb2430db8435/lib/debugbar/subscribers/active_record.rb:29:in `each'
	from .rvm/gems/ruby-3.3.0/bundler/gems/debugbar-eb2430db8435/lib/debugbar/subscribers/active_record.rb:29:in `each_with_index'
	from .rvm/gems/ruby-3.3.0/bundler/gems/debugbar-eb2430db8435/lib/debugbar/subscribers/active_record.rb:29:in `sql'

and I get the same error when trying to process a request from rails s
I also tried the tagged version and master from the repo, with the same results as from the rubygems version.
let me know if there's any other information that could be helpful.  Thanks, and interested in trying this out!

Can you try v.0.1.4 please 🙏
I'm really sorry for this, it was silly to try to fix it quickly instead of removing it in #7.


[DEVHAS-213] Properly handle CDQs for repo URL with tailing '/'

Signed-off-by: John Collier<EMAIL_ADDRESS>What does this PR do?:
This PR updates the logic for the component name generation that CDQs use to properly handle Git repositories that end with a tailing forward slash (e.g. https://github.com/devfile-samples/devfile-sample-java-springboot-basic/). Previously, an empty string would be returned, instead of the Git repository name, leading to an invalid Component name like -ohu7.  With this change, we now properly handle URLs with a tailing slash. Furthermore, if sanitizeComponentName is called with an empty name, we use the gofakeit package to generate an empty string.
Additionally, I've updated the logic setting the status on CDQs, where if the status update fails, we try again by only updating the status condition.
Which issue(s)/story(ies) does this PR fixes:
https://issues.redhat.com/browse/DEVHAS-213
PR acceptance criteria:

[x] Unit/Functional tests

[ ] Documentation

[ ] Client Impact

How to test changes / Special notes to the reviewer:

/retest


How to remove black bacground of output image

After running the model and removal background, the output result has black background. My question is how to get output image without black, i mean save output image (image_preds[0]) as transparent image.
The code:
def predict(self, image) :
        images = [image]
        image_shapes = [image.shape[:2] for image in images]
        images = [array_to_pil_image(image, self.resolution) for image in images]
        image_preprocessor = ImagePreprocessor(resolution=self.resolution)
        images_proc = []
        for image in images:
            images_proc.append(image_preprocessor.proc(image))
        images_proc = torch.cat([image_proc.unsqueeze(0) for image_proc in images_proc])

        with torch.no_grad():
            scaled_preds_tensor = self.model(images_proc.to(self.device))[-1].sigmoid()
        preds = []
        for image_shape, pred_tensor in zip(image_shapes, scaled_preds_tensor):
            if self.device == 'cuda':
                pred_tensor = pred_tensor.cpu()
            preds.append(torch.nn.functional.interpolate(pred_tensor.unsqueeze(0), size=image_shape, mode='bilinear', align_corners=True).squeeze().numpy())
        image_preds = []
        for image, pred in zip(images, preds):
            image = image.resize(pred.shape[::-1])
            pred = np.repeat(np.expand_dims(pred, axis=-1), 3, axis=-1)
            image_preds.append((pred * image).astype(np.uint8))

        return image, image_preds[0]

Here, you get the returned value of both the predicted mask and the original image. With the mask (0 or 1), you can use mask * original image + (1 - mask) * new background to replace the original background with a new image.


libsForQt5.mlt: 7.22.0 -> 7.24.0

Automatic update generated by nixpkgs-update tools. This update was made based on information from passthru.updateScript.
meta.description for libsForQt5.mlt is: Open source multimedia framework, designed for television broadcasting
meta.homepage for libsForQt5.mlt is: https://www.mltframework.org/
Updates performed

Ran passthru.UpdateScript

To inspect upstream changes

Release on GitHub

Compare changes on GitHub

Impact
Checks done

built on NixOS
The tests defined in passthru.tests, if any, passed
found 7.24.0 with grep in /nix/store/2g2xvi7lj162bl72hnxm8g9d16zsrrk6-mlt-7.24.0
found 7.24.0 in filename of file in /nix/store/2g2xvi7lj162bl72hnxm8g9d16zsrrk6-mlt-7.24.0

Rebuild report (if merged into master) (click to expand)

17 total rebuild path(s)

17 package rebuild(s)

First fifty rebuilds by attrpath

flowblade
hikounomizu
kdePackages.kdenlive
kdePackages.mlt
krita
krita-plugin-gmic
libsForQt5.kdenlive
libsForQt5.mlt
mlt
plasma5Packages.kdenlive
plasma5Packages.mlt
python311Packages.mlt
python312Packages.mlt
qt6Packages.mlt
shotcut
synfigstudio

Instructions to test this update (click to expand)

Either download from Cachix:
nix-store -r /nix/store/2g2xvi7lj162bl72hnxm8g9d16zsrrk6-mlt-7.24.0 \
  --option binary-caches 'https://cache.nixos.org/ https://nix-community.cachix.org/' \
  --option trusted-public-keys '
  nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(The Cachix cache is only trusted for this store-path realization.)
For the Cachix download to work, your user must be in the trusted-users list or you can use sudo since root is effectively trusted.
Or, build yourself:
nix-build -A libsForQt5.mlt https://github.com/r-ryantm/nixpkgs/archive/f858ba78b0992ed19b984ef2297335599af0d3d9.tar.gz

Or:
nix build github:r-ryantm/nixpkgs/f858ba78b0992ed19b984ef2297335599af0d3d9#libsForQt5.mlt

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/2g2xvi7lj162bl72hnxm8g9d16zsrrk6-mlt-7.24.0
ls -la /nix/store/2g2xvi7lj162bl72hnxm8g9d16zsrrk6-mlt-7.24.0/bin

Pre-merge build results
We have automatically built all packages that will get rebuilt due to
this change.
This gives evidence on whether the upgrade will break dependent packages.
Note sometimes packages show up as failed to build independent of the
change, simply because they are already broken on the target branch.
Result of nixpkgs-review --extra-nixpkgs-config '{ allowInsecurePredicate = x: true; }' run on x86_64-linux 1

1 package failed to build:

pixinsight

20 packages built:

flowblade
hikounomizu
kdePackages.kdenlive
kdePackages.kdenlive.debug
kdePackages.kdenlive.dev
kdePackages.mlt
kdePackages.mlt.dev
krita
krita-plugin-gmic
libsForQt5.kdenlive
libsForQt5.mlt
libsForQt5.mlt.dev
mlt
mlt.dev
python311Packages.mlt
python311Packages.mlt.dev
python312Packages.mlt
python312Packages.mlt.dev
shotcut
synfigstudio

Maintainer pings
cc @cillianderoiste for testing.

Already on master.


Examples viewable in pre-executed notebook

I'm looking to use movingpandas on a project and to get a feel for the functionality looking at the examples really helps, however I don't really want to run all the examples, just have a scan/search through them and the output for what I'm after. To help with this, it would be useful to have an executed version of the examples saved somewhere, either the notebooks themselves, or them in markdown/html.

Welcome, Rachel! I totally agree that we should have a rendered copy of the notebooks accessible somewhere. I haven't found a good (hopefully automated) solution yet, however.
The rendered notebooks get quite large, particularly when holoviz graphics are involved. Therefore, I want to avoid adding the rendered notebooks directly to this repo. If you know of a solution for this issue, any recommendations are welcome.

Hello! Thanks for the speedy response (and for the library, looking forward to having a play).
Hmm I'm not sure. Could look at using https://nbautoexport.drivendata.org/stable/ to autogenerate the markdown (or html) - perhaps could create a pipeline to run and add the output somewhere else. I sometimes keep the output in my repos to help with pull requests, but might not help with the size issue that much unless can pipe it somewhere else.

I've created a new repo https://github.com/anitagraser/movingpandas-website to play around with and added badges that link to the rendered notebook htmls:

Nice one - thanks!

The analysis examples should all be viewable now.

Awesome! Thank you!


Permission Sets

Last time (Extensions 1.0) I had to compile a permissions.xml in my NAVX file with the permission set of my app. Is this still needed, if so, how?

You can simply use the same XML file you generated for your V1 extension, or use the same process to make a new one for your V2 extension, and then put the XML file into your extension project. Seems like you can put it anywhere you want and the compiler will figure out what to do with it.
For example, I put mine in a specific folder, and no other settings were needed to be changed and the compiler picked it up and included it in the compiled extension app.

Ahhh... "The file matches the definition for 'PermissionSet'.". Thank you very much!

I'm just wondering what object type page extensions and table extensions are? (  ??). I searched here, I didn't find anything..

Just give permissions to the original table/page objects you are extending, not the actual extension objects.


chat/instruct task type

A nice addition to this tool would be a chat/instruct type would generate a longer more detailed prompts (based on some initial guidance on the task). I tried text_gen and NLI on some unstructured data but neither returned a suitable chat/instruct style input/output.

Thanks for the suggestion!


Multiple games without PGN tags don't get imported by most software, including Lichess studies

A very simple PGN tag will solve this, but is there some other method?

fixed in v2.3.133


Automatic sticky bucketing for logged out users if ID is unset

How do you handle bucketing for users that are logged out and keep them sticky so they don't see a different variation on each page load?
Optimizely doesn't work unless submit a user id when setting up the provider.
I have an oauth 2.0 pkce login flow so I don't know about a logged in user until the client has access to window.
I'm currently assigning them a random ID, if I have nothing in user state or local storage and saving it to local storage but that is multiple re-renders when using a provider the wraps the entire app on the first visit.  it's really nasty.
It would be nice if when no user id was provided Optimizely just handled it on client initialisation with ip/user agent hash or something. So for useDecision stuff for Optimizely full stack can do what Optimizely web can do without user IDs, I can't do this sort of stuff from the client. I could handle a single re-render on log in. This happens automatically for Optimizely web and ironically Optimizely full stack when I'm using flags it doesn't even though it costs a lot more money.

The suggested option with IP / user agent does not sound like valid / achievable option, because during SSR it is not possible to determine such values by from the react components tree.

How do you handle bucketing for users that are logged out and keep them sticky so they don't see a different variation on each page load?

You can create a cookie with a random unique value, that will stay in the customer browser for years (i.e. optimizelyEndUserId so that both WEB and FullStack can use the same user ID). You can use IP or user agent as you suggested, that will be provided to react by your Node.js server during SSR, or you can use any other approach that suit your needs.

I have an oauth 2.0 pkce login flow so I don't know about a logged in user until the client has access to window.
I'm currently assigning them a random ID, if I have nothing in user state or local storage and saving it to local storage but that is multiple re-renders when using a provider the wraps the entire app on the first visit. it's really nasty.

This sound like an issue with the way you integrated your application with Optimizely react-sdk, because it should not happen.

import { FC } from "react";
import { OptimizelyProvider, ReactSDKClient } from "@optimizely/react-sdk";
import { getOptimizelyInstance } from "src/services/optimizely";
import { useFundraiserContext } from "src/contexts/FundraiserContext";
import { isBrowser } from "src/utils/browserUtils";
import { useLocalStorage } from "src/hooks/useLocalStorage";
import useLayoutEffectBrowser from "src/hooks/useLayoutEffectBrowser";

const ENV_NAME = process.env.NEXT_PUBLIC_ENV_NAME || "";

const uuidv4 = (): string => {
  return "xxxxxxxx-xxxx-4xxx-yxxx-xxxxxxxxxxxx".replace(/[xy]/g, (c) => {
    var r = (Math.random() * 16) | 0,
      v = c == "x" ? r : (r & 0x3) | 0x8;
    return v.toString(16);
  });
};

export const OptimizelyWrapper: FC<{}> = ({ children }) => {

  const client: ReactSDKClient = getOptimizelyInstance();
  const [fundraiserState] = useFundraiserContext();
  const { fundraiser } = fundraiserState;

// first prop is the key name and the second is the default before window is ready
  const [optimizelyId, setOptimizelyId] = useLocalStorage(
    "optimizely",
    "logged_out"
  );

  useLayoutEffectBrowser(() => {
    if (optimizelyId === "logged_out" && !fundraiser?.uniqueId)
      setOptimizelyId(uuidv4());
  }, []);

  const userID = fundraiser?.uniqueId || optimizelyId;
  // For test stability
  const isDev = ENV_NAME === "development";

  return (
    <>
      {client && !isDev ? (
        <OptimizelyProvider
          optimizely={client}
          user={{
            id: userID,
            attributes: {
              user_id: userID,
              is_logged_in: !!fundraiser?.uniqueId,
            },
          }}
          isServerSide={!isBrowser}
        >
          {children}
        </OptimizelyProvider>
      ) : (
        <>{children}</>
      )}
    </>
  );
};

export default OptimizelyWrapper;

Since you are using SSR (next.js?), you should not use local storage, because it is not accessible on the server and you will have different output during SSR and CSR which might break hydration process. You should consider switching to cookies. You can take a look at this repository: https://github.com/oMatej/nextjs-optimizely.
It has the issue I described here: https://github.com/optimizely/react-sdk/issues/87, but there is a chance it got fixed in recent updates of the SDK. I also described there potential workaround, but I am not sure if this still is a thing with the latest version of @optimizely/react-sdk.
What it does:

Fetches the datafile during SSR so that Optimizely can generate proper HTML
Generate or reuse previous user id from the cookies
Passes the datafile and id through getServerSideProps to the client
Uses these values to set up new optimizely client instance in the browser

Thanks @oMatej I will have look at implementing cookies.
How important is datafile? When using things like useDecision its user/bucket specific, the the datafile from server looks like more general info for feature switching, could it be be blank and would useDecision still work?
With regards to user ID I'm pretty much generating it as you describe. The trick it going to get that stuff accessible server side.

How important is datafile? When using things like useDecision its user/bucket specific, the the datafile from server looks like more general info for feature switching, could it be be blank and would useDecision still work?

The datafile contain all the data related to Optimizely experiments / features you configure in the dashboard, so it is kinda crucial. If your application support SSR then you need to fetch it on your own according to this: https://github.com/optimizely/react-sdk#server-side-rendering.
Providing sdkKey during initialisation of Optimizely instance would fetch the datafile for you, but it will not work during SSR (so it would cause different content rendered on the server, and then after rendering in the browser it would change causing flickering for the users which is not desired).

I if the datafile is crucial why does the async loading example on the main readme doesn't not even bother with it.
Unfortunately the user ID from cookies back into the response request isn't going to work in my use case because that is going to screw up my caching strategy. All pages have a logged in and logged out view, logged in data happens async and for most people viewing the pages they will see the highly cached logged out view. If I start passing cookies down with the request it is going to break caching.

Why do I even have to instantiate a client why can't I just pass my SDK key into the provider and let the provider handle everything?

I if the datafile is crucial why does the async loading example on the main readme doesn't not even bother with it.

Of course it does. It just fetch it asynchronously based on the provided sdkKey. When the application does not support SSR, it initially render just a blank page, so Optimizely can prevent rendering this or another component untill it finish fetching datafile (or the timeout will be triggered). You do not have flickering of two variations in such a case, because it can just wait with rendering the element of the page until it has the information about selected variant.

Why do I even have to instantiate a client why can't I just pass my SDK key into the provider and let the provider handle everything?

During SSR your goal should be to render exactly the same content on the server, and in the browser when the hydration process is happening. That is why react logs errors when this is not happening.
You can initialize Optimizely only on the client side. However, it might cause errors with react hydration process, and it will not be the nicest user experience because i.e. the button will be rendered with orange color on the server and on the initial load in the customer browser, and then, X ms later, it will switch color to green because Optimizely got initialized asynchronously in the browser and the customer got assigned to the experiment.
It is the same thing as doing SSR with i.e. redux or any other state management library. If you want to prefill the store with some data, you need to gather them before sending the HTML to the client.

I think my first problem is that the Provider component is completely useless in my use case because it is so rigid I wish the provider looked more like this:
import React, {
  FC,
  useState,
  createContext,
  useContext,
  Dispatch,
  SetStateAction,
} from "react";

import { ReactSDKClient } from "@optimizely/react-sdk";

export type OptimizelyContextStateType = {
  optimizely: ReactSDKClient | null;
  isServerSide: boolean;
  timeout: number | undefined;
};

const defaultState: OptimizelyContextStateType = {
  optimizely: null,
  isServerSide: false,
  timeout: 0,
};

export type OptimizelyContextProviderType = [
  OptimizelyContextStateType,
  Dispatch<SetStateAction<OptimizelyContextStateType>>
];

export const OptimizelyContext = createContext<OptimizelyContextProviderType>([
  { ...defaultState },
  () => {},
]);

export const useOptimizelyContext = () => useContext(OptimizelyContext);

export const OptimizelyContextProvider: FC = ({ children }) => {
  const [state, setState] = useState({
    ...defaultState,
  });

  return (
    <OptimizelyContext.Provider value={[state, setState]}>
      {children}
    </OptimizelyContext.Provider>
  );
};

No user ID stuff at all, and see the setter is exposed
Then you can choose how you set the user for you could do directly in the component so it renders SSR or I could do something like this to it works client side
export const OptimizelyLoader: FC<{}> = ({ children }) => {
  const { trackError } = useTrackingContext();
  const [optimizelyData, setOptimizelyData] = useOptimizelyContext();
  const { optimizely } = optimizelyData;
  const [fundraiserState] = useFundraiserContext();
  const { fundraiser } = fundraiserState;

  // "logged_out" is default effectively user_id unset
  const [optimizelyId, setOptimizelyId] = useLocalStorage(
    "optimizely",
    "logged_out"
  );

  const uuidv4 = (): string => {
    return "xxxxxxxx-xxxx-4xxx-yxxx-xxxxxxxxxxxx".replace(/[xy]/g, (c) => {
      var r = (Math.random() * 16) | 0,
        v = c == "x" ? r : (r & 0x3) | 0x8;
      return v.toString(16);
    });
  };

  const getClientInstance = async () => {
    try {
      const resp = await fetch(
        `https://cdn.optimizely.com/datafiles/${OPTIMIZELY_SDK_KEY}.json`
      );
      const datafile = await resp.json();
      const instance = createInstance({
        datafile: datafile,
      });
      return instance;
    } catch (err) {
      trackError(err as Error);
      return null;
    }
  };

  useLayoutEffectBrowser(() => {
    if (!OPTIMIZELY_SDK_KEY.length) return;

    if (optimizelyId === "logged_out" && !fundraiser?.uniqueId) {
      setOptimizelyId(uuidv4());
    }
    getClientInstance().then((client: ReactSDKClient | null) => {
      if (!!client) {
        setOptimizelyData({
          ...optimizelyData,
          optimizely: client,
        });
      }
    });
  }, []);

  useEffect(() => {
    const userID = fundraiser?.uniqueId || optimizelyId;

    if (userID && optimizely) {
      const userData = {
        id: userID,
        attributes: {
          user_id: userID,
          is_logged_in: !!fundraiser?.uniqueId,
        },
      };

      optimizely.setUser(userData);
    }
  }, [optimizely, optimizelyId, fundraiser]);

  return <></>;
};

export default OptimizelyLoader;

Hello All,
I'm working to clean up the latent GitHub Issues for the React and Javascript repos.  I know we're way behind here and thanks for handing in there with us as we optimize our processes.
After reading through this one. I wonder if the Advance Audience Targeting's (AAT) VUID would work to accomplish sticky bucketing for a logged out user.

We released v3 of the React SDK allowing for anonymous instantiation of the optimizely client without a user.id in client-side contexts. Versions > 3.0.1 will have this new paradigm ironed out.
linked to internal ticket FSSDK-9985


Before Hook guard and currentRoute fullPath

Version
3.4.8
Reproduction link
https://jsfiddle.net/9oxdastu/
Steps to reproduce

Implement a beforeEach hook on the router
On the beforeHook use a next({ name: 'PageNotFound' })
Call a router.push() and expect that it use the previous line next()
Using a catch on the router.push that should be redirected, does not updates router.currentRoute.fullPath to match the PageNotFound URL
After the block execute router.push().then.catch, even if I sleep(500), the router.currentRoute.fullPath is not updated correctly.
When I use the afterHook on the router, I Can correctly see my router updated

What is expected?
I expect the "router.currentRoute.fullPath" to match the route I redirect to in the beforeEach hook next()
What is actually happening?
The router.currentRoute.fullPath doesn't get updated, it reflects the endpoint prior to the redirected router.push()

The reproduction seems incomplete, it's not runnable. Please fix the reproduction to it shows the actual problem when running it.

Remember to use the forum or the Discord chat to ask questions! If you manage to boil down the reproduction open a new issue


Question: may i have a solution for this please.

since we can run linux/aarch64 binaries on termux now because of glibc for Termux as long as there are no hardcoded paths i tried to use sdkman for installing openjdk-21 but it fails to extract the zip file after downloading completes. it seems sdkman is expecting linux environment for the shell. i probably don't know what i'm saying since i have recently got into this programming world. please i need a solution. yes Termux already has bionic based openjdk-17 available but i want to use newer versions since i can run glibc based ELFs on termux.
java and sdkman versions:

where it has failed:

i think it's java problem. i tested with gradle and kotlin.  why only java?

please change /usr/bin/env tar and /usr/bib/env zip to $PREFIX/bin/env tar and $PREFIX/bin/env zip in __sdkman_post_installation_hook  function. it will work both in FHS compliant and non FHS compliant environments i guess and my problem will be solved.

i'm not having this problem with other candidates that don't have post_installation_hook related hardcoded paths like kotlin.


Using and adapting "inline" mesh files

Hello,
I am looking to adapt the "inline" family of mesh files (specifically, "inline_tri.mesh" and "inline_tet.mesh") to add more attributes for the elements of the mesh. I am trying to create a PWConstCoefficient k for a DiffusionIntegrator that changes values in some subset of the original domain, and it seems to me that the simplest way to do this is to have a Vector of size mesh->attributes.Max() with different values of k for each entry, and then feed this into the PWConstantCoefficient constructor.
However, that set of mesh files is not written in the standard mesh v1.0 format, so there is no marking  of any attributes.  Is there any way to either:
1.) Change the "inline-tri.mesh" file itself to assign attributes for the mesh?
or
2.) Change the attributes of the elements in my example file before the mesh is refined to mark the elements I need to with the given attribute,
Of course, I could also define a piecewise constant function defining k on the different parts of the domain and feed it into a FunctionCoefficient to feed into the DiffusionIntegrator, or create my own mesh file with the desired attributes, but I figured I should at least ask, since editing this file would be the easiest way to adjust for what I need.
Thanks in advance!

You can set the attributes of the elements (or boundary element) in a Mesh using:
   mesh.GetElement(elem_id)->SetAttribute(attr);
   mesh.GetBdrElement(bdr_elem_id)->SetAttribute(bdr_attr);

or in the current development version of mfem you can use:
   mesh.SetAttribute(elem_id, attr);

After updating the attributes of the elements (and/or boundary elements), you should call:
   mesh.SetAttributes();

which will update the arrays mesh.attributes and mesh.bdr_attributes.
You can also take a look at the new shaper.cpp miniapp which performs some attribute modifications among other things.

Hello,
Just wanted to let you know, this works just fine; I was able to define the PWConstCoefficient correctly using this, and it works even after refining.
Now I am wondering which is better to use: a PWConstCoefficient or a FunctionCoefficient?  I have been testing the two just now and I find that when the rhs is 0 (i.e, I solve for a linear or constant function), the FunctionCoefficient performs better (i.e, the error is less), but when I try to solve for exact solution with nonzero rhs (quadratic), they both perform around the same, with FunctionCoefficient having bigger error for some dofs, but PWConstCoefficient having bigger error for others.
It should be noted that due to the piecewise constant nature of the coefficient, neither solves well, since the rhs I compute isn't exact (I just pull out the K(x) and multiply by div(\grad p(x))), so the error is larger than expected on DOFs on the boundary where K changes.

I'd expect that both PWConstCoefficient and FunctionCoefficient to give the same result, if they define the same function with jumps aligned with the mesh. I'm not sure why you get different solutions. Regardless, I don't have a recommendation which one is better to use - it is a matter of personal preference, I guess. However, it may be good to understand why they give you different results - that may tip the scales one way or the other.

Hello!
Just wanted to let you know, you were correct, they give the same result; the difference was from my implementation of the right hand side.  When I looked at the the global stiffness matrix itself though, the two were identical (at least, in the coarsest 2D case (no refinements).  I haven't checked the global stiffness matrix on more refined meshes, but if they're the same on the coarsest, I see no reason why the finer ones would differ.

Glad you got it working @jxb3641. I'm closing this issue now. Feel free to reopen if you have any further questions.


SimulationPacketLossChance works wrong

Hi!
To check if packet should be lost there is following code:
_randomGenerator.Next(100/SimulationPacketLossChance) == 0
This is obviously a bad idea:

SimulationPacketLossChance might be zero
if SimulationPacketLossChance is 51 or higher, probability of getting 0 equals 1, so the packet will be dropped.

The simplest way around is to do exactly what probability in percentages is:
_randomGenerator.Next(101) < SimulationPacketLossChance
I put 101 here, because else probability of 99 percent would trigger every time. This is quite obviously not an accurate solution, probability of one percent here is actually 1/101 instead of 1/100. But it is because of using discrete values. Another option would be:
(_randomGenerator.NextDouble() * 100 < SimulationPacketLossChance
This is obviously an accurate way, but a soon as I know it is slower. Although it shouldn't be a problem, since this code executes for debug purposes only.

Please tell me what you think:)
I would like to add a pull request, but i'm not sure what is the right way to contribute in this project. Is there anywhere a document?

@marsermd Hi! Yes there is some bad code) You can add pull request. Just use code style like in all project)

@RevenantX Thank you for fast response!
What is your branching policy? Should I create a separate branch?
Also I'd like to add Put(char c) and GetChar() to NetDataWriter and NetDataReader respectfully. Should I create issue for that or I can proceed to creating a pull request?

Do it in master branch. And add char methods if you want in that pull request)

It took me longer to setup VS to run tests then to write code:D
But now I can contribute :)
Looking forward to further collaboration.


Refactor into a composite action

WIP.
Closes #8.

This works, but some open questions:

How do composite actions interact with their parent workflow? I don't have to install python or any other state here, presumably because I'm running in the ubuntu-latest instance defined in the parent workflow, but can I rely on this? Should the action opportunistically use actions/setup-python if it can't detect a modern enough Python?
How does this interact with the target repository's working directory? Normally it's the $CWD after an actions/checkout, but we don't guarantee that here.
How should this interact with virtual environments? Right now this action is completely unaware of them (and was before as well)

How do composite actions interact with their parent workflow? I don't have to install python or any other state here, presumably because I'm running in the ubuntu-latest instance defined in the parent workflow, but can I rely on this? Should the action opportunistically use actions/setup-python if it can't detect a modern enough Python?

I think the answers to this are "technically no, but yes" and "depends on the first."
In principle it's a user error to attempt to run pip-audit on a host that doesn't have Python installed, since there's no Python environments whatsoever to audit. But maybe that's too legalistic of us, since someone might want to audit a fully-resolved and hashed requirements file without worrying about the rest of the Python environment? We should probably support that case.
If the first is "no", then the second is definitely "yes."


How to remove prefix in people picker after configure AzureCP

Hi Yvand,
Hope you already have this question.
Do you know how do i remove in the people picker the claims prefix after we configure the AzureCP?

Thank you for you attention.

Hi @aaclage, no, it cannot be removed.
Note that:

Only the results that match the identity claim type don't have any prefix
For the others, the text displayed, e.g. "Group", comes from the property DisplayName returned by (Get-SPTrustedIdentityTokenIssuer "TRUSTNAME").ClaimTypeInformation

Hi @Yvand
Thank you for your response and support ,
I was confuse with your first answer "Only the results that match the identity claim type don't have any prefix",
Base on the claim type it's always define for "user", since groups cannot be incorporated in current, added new claim for groups.
Because of that "(Groups)" prefix will appear, this statement is correct?
Bellow my configuration table:

@aaclage correct, this screenshot shows 2 claim types, and only the role results will show the prefix

Hi @Yvand
It's a little annoying we are not able to use one claim for both (user & group) to retrieve both and prefix disappear but understand the technical reason.
I will close this ticket.
Thank you very much for your support,


Update project

This PR updates this project:

Kotlin v2.0.0
Compile SDK to 34
AGP to v8.5.0
Compose Gradle Plugin v2.0.0
Gradle wrapper v8.8
Upgrade all other dependencies
Replace Accompanist with Compose Foundation APIs
Upgrade Exoplayer APIs to latest version
Remove more deprecated code
Added editorconfig file to handle Composable functions in IDE when using KtLint Plugin
Cleanup build.gradle files, extract common config into common-config.gradle file

Regarding this:

Open Question:
I switched to latest Exoplayer which is part of Media3 libraries.
Without knowing what the bug was, I cannot declare it is fixed in the current version. Readme in this project doesn't link to the right question anymore and the link is broken. This is why I have not removed it from Readme yet.
If the original author of the lib can verify the bug is fixed, I can also remove this info from  Readme in this PR :)

Thank you very much for your contributions.


feat(plotly): Bump plotly from 1.25.0 to 1.41.1

Closes #5.

/cc @rgbkrk

Fantastic, thank you!

Want to ship it too? If it's your first time publishing on npm, I can walk you through it.

Sure, I can do that 👍.  I haven't shipped anything using semantic-release specifically, but I have published ion and create-nteract-app to npm.
It seems like travis is setup to run the semantic-release script for us, but how do we trigger travis to do that? I would have expected it to happen once a commit is made to master (like the merging of this PR), but it doesn't seem like that happened.

You know, I'm not sure. @lgeiger set this up.

Looks like the automatically publishing is failing https://travis-ci.org/nteract/minimal-plotly/builds/430590821#L1727

Probably due to the semantic release upgrade: https://github.com/nteract/minimal-plotly/pull/6/commits/025bc1269b37726145d5971196f7a9e33f6bc171

I can setup a new auto-publishing tomorrow, or we can switch back to manual publishing and rip out semantic-release.

@lgeiger Any chance you have time to setup the auto-publish?


add retry interval (base_interval, max_interval)

Added envoy-filter for retry interval.
Added description of retry interval to README.md

@niqniqniqq
This is all comment from me.
Please check it. Good commit!


db dump tool: include sql file extension

Description
The .editorconfig contains
[*.sql]
trim_trailing_whitespace = false

Since this is rather important it would be helpful if the generated file already has the appropriate extension. As a small bonus it enables syntax highlighting.
Self Check:
Strike through any lines that are not applicable (~~line~~) then check the box

[ ] Attached issue to pull request
[ ] Changelog entry
[ ] Type annotations are present
[ ] Code is clear and sufficiently documented
[ ] No (preventable) type errors (check using make mypy or make mypy-diff)
[ ] Sufficient test cases (reproduces the bug/tests the requested feature)
[ ] Correct, in line with design
[ ] End user documentation is included or an issue is created for end-user documentation (add ref to issue here: )

Reviewer Checklist:

[ ] Sufficient test cases (reproduces the bug/tests the requested feature)
[ ] Code is clear and sufficiently documented
[ ] Correct, in line with design

Processing this pull request

Merged into branches master in 972add51351edd2ccdadc65b005980d8b0a2fb61

Processing #4704.

Processing #4703.


MGMT-19058: Add OpenShif AI operator

This patch adds to the operators page of the cluster creation wizard a new check-box that will be used to enable the installation of the OpenShift AI operator.
https://github.com/user-attachments/assets/e0cd543b-89fe-4e3e-8ade-813dda350dfb
Related: https://issues.redhat.com/browse/MGMT-19058
Related: https://issues.redhat.com/browse/MGMT-19057

Added a commit to improve the display of requirements and dependencies:


Add highlight of text frames on search results

Search results are displayed without any means, only ids are showed up.
Adds highlight of text frames matched would improve usability]
Solution:

dynamically configure fields to be highlighted by Elastic
remove custom JSON format for search results and facets tree in favour of ES format

Done. Closing.


Mar 731 refactor signup page

General comment - should we do some unit tests?

Yeah unit tests for sure, but maybe once the whole page is completed


[SPARK-14498][ML][PYTHON][SQL] Many cleanups to ML and ML-related docs

What changes were proposed in this pull request?
Cleanups to documentation.  No changes to code.

GBT docs: Move Scala doc for private object GradientBoostedTrees to public docs for GBTClassifier,Regressor
GLM regParam: needs doc saying it is for L2 only
TrainValidationSplitModel: add .. versionadded:: 2.0.0
Rename “_transformer_params_from_java” to “_transfer_params_from_java”
LogReg Summary classes: “probability” col should not say “calibrated”
LR summaries: coefficientStandardErrors —> document that intercept stderr comes last.  Same for t,p-values
approxCountDistinct: Document meaning of “rsd" argument.
LDA: note which params are for online LDA only

How was this patch tested?
Doc build

CC @dbtsai Could you please confirm the doc changes about coefficientStandardErrors and GLM's regParam?  Thanks!

Test build #55394 has started for PR 12266 at commit e746a17.

Test build #55394 has finished for PR 12266 at commit e746a17.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/55394/
Test FAILed.

Test build #55395 has started for PR 12266 at commit 56acf90.

Test build #55395 has finished for PR 12266 at commit 56acf90.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/55395/
Test PASSed.

Both looks good to me. Thanks. I'll go ahead and merge it soon.

Thanks. Merged into master.

Thanks for taking a look!


check travis keys for Lasso with namechange

The name change seemed to have broken our travis deployment to ghpages.   Can you check that this isn't something on the Travis side?

I do not see a .readthedocs.yml in the Lasso repository. Could that be the problem?

nope. we aren't using readthedocs
On Wed, Jan 8, 2020 at 12:39 PM David Ory<EMAIL_ADDRESS>wrote:

I do not see a .readthedocs.yml in the Lasso repository. Could that be
the problem?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/wsp-sag/Lasso/issues/47?email_source=notifications&email_token=AAGCBI2O2LMT454CBUL57NTQ4Y2ZXA5CNFSM4KEACPVKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEIN4YCI#issuecomment-572247049,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAGCBIZAQ4HHH45PJO6UE5TQ4Y2ZXANCNFSM4KEACPVA
.

Ah, okay. So just GitHub pages. I don't see that deployment in the .travis.yml. I would expect to see something like:
deploy:
  provider: pages
  skip_cleanup: true
  github_token: $GITHUB_TOKEN  # Set in the settings page of your repository, as a secure variable
  keep_history: true
  on:
    branch: master

How are you telling Travis to build it?

Its on develop branch and currently  deploys based on develop:
deploy:
provider: pages
skip_cleanup: true
local_dir: docs/_build/html
github_token: $GITHUB_TOKEN # Set in the settings page of your repository,
as a secure variable
keep_history: true
on:
branch: develop # will change to master when we have it figured out ```

On Wed, Jan 8, 2020 at 4:08 PM David Ory<EMAIL_ADDRESS>wrote:

> Ah, okay. So just GitHub pages. I don't see that deployment in the
> .travis.yml. I would expect to see something like:
>
> deploy:
>   provider: pages
>   skip_cleanup: true
>   github_token: $GITHUB_TOKEN  # Set in the settings page of your repository, as a secure variable
>   keep_history: true
>   on:
>     branch: master
>
> How are you telling Travis to build it?
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/wsp-sag/Lasso/issues/47?email_source=notifications&email_token=AAGCBIY75FR5UBFLQMEPGRDQ4ZTIXA5CNFSM4KEACPVKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEIOOGHQ#issuecomment-572318494>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AAGCBI4CMCXKIMR7HXWK5HDQ4ZTIXANCNFSM4KEACPVA>
> .
>


Missing Report for jar

jar_ana.zip
The attached zip has 5 jars. By running report for this in ODC version 7.4.1 will get only result for 4 jars.
Jars In a folder :
jedis-4.3.1.jar
dd-plist-1.8.jar
spring-core-5.3.24.jar
spring-beans-5.3.23.jar
spring-context-5.3.23.jar --->  Not getting result for this jar
But if the 'spring-context-5.3.23.jar' moved into a subfolder inside a main folder then will get report for all 5. I don't know why?
Explain what is the issue  here?

Take a closer look at your report.... there's a section 'related dependencies' That's Dependency-check seeing that spring-beans and spring-context are both part of the Spring Framework version 5.3.23. Were you to use spring-core-5.3.23.jar instead of 5.3.24 then all three spring jars would be accounted for under a single entry


Discuss WIP pull requests

As this already came up in a real world discussion today, may be we should prioritize adding some details about this.

I agree - it is a useful mechanism to get feedback on unfinished code.


[WIP] Test skip compile

jenkins, test this with java 11 please

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Alluxio-PR-Builder/10962/
Test PASSed.

alluxio-bot, merge this please


Centralize the initContainer that fixes ownership

We were copy-pasting this, and it was causing shared-readwrite
to not be functional in some cases.
Fixes https://github.com/2i2c-org/infrastructure/issues/440

Nice

I'll merge this one in since it has a few approvals

@yuvipanda, is there anything else you want to add here, otherwise, I think it is ready...

@damianavila ya just wanted to find a time when I've a few hours uninterrupted in case something goes wrong. Hopefully tomorrow.


ran the mps benchmark on m1 max (32GPU)

Hi,
I'm not sure if you are open to more PRs for these test, but here is one. I can also run this on my m1 ultra. Thanks for creating the benchmark.

Thanks! To keep things simple, I will probably only update the VGG results from now on. In particular, the Ultra results may be interestin

g

Ok, I've added the m1ultra results for vgg. Cheers!


Engine API: proposal for authentication

This is an RFC for authenticated JSON rpc API
Authentication
The engine JSON-RPC interface, exposed by EL and consumed by CL, needs to be authenticated. The authentication scheme chosen for thus purpose is JWT.
The type of attacks that this authentication scheme attempts to protect against are the following:

RPC port exposed towards the internet, allowing attackers to exchange messages with EL engine api.
RPC port exposed towards the browser, allowing malicious webpages to submit messages to the EL engine api.

The authentication scheme is not designed to

Prevent attackers with capability to read ('sniff') network traffic from reading the traffic,
Prevent attackers with capability to read ('sniff') network traffic from performing replay-attacks of earlier messages.

Authentication is performed as follows:

For HTTP dialogue, each jsonrpc request is individually authenticated by supplying JWT token in the HTTP header.
For a WebSocket dialogue, only the initial handshake is authenticated, after which the message dialogue proceeds without further use of JWT.
For inproc, a.k.a raw ipc communication, no authentication is required, under the assumption that a process able to access ipc channels for the process, which usually means local file access, is already sufficiently permissioned that further authentication requirements do not add security.

JWT specifications

The EL MUST support the at least the following alg HMAC + SHA256 (HS256)
The EL MUST reject the alg none.

The HMAC algorithm means that symmetric encryption is used, thus several CL's will be able to use the same key, and, from an authentication perspective be able to impersonate each other. From a deployment perspective, it means that an EL does not need to be provisioned with individual keys for each caller.
Key distribution
The EL and CL clients MUST accept a cli/config parameter: jwt-secret, a 256 bit key, to be used for verifying/generating jwt tokens.
If such a parameter is not given, the client SHOULD generate such a token, valid for the duration of the execution, and show the token in the output, which the user can then use to provision the counterpart client with.

Would you mind to provide more details and examples of exact payloads during the handshake for the WebSocket case? It seems like we need to add a new error code that would signal the caller that it's not authenticated.

The websocket handshake starts with the client performing a websocket upgrade request. This must be a regular http GET request, and the actual
parameters for the WS-handshake is carried in the http headers.
Example:
GET / HTTP/1.1
Host: localhost:1337
User-Agent: Mozilla/5.0..
Origin: ..
Connection: keep-alive, Upgrade
Upgrade: websocket
Sec-WebSocket-Version: 13
Sec-WebSocket-Protocol: echo-protocol
Sec-WebSocket-Extensions: permessage-deflate
Sec-WebSocket-Key: VOPn9g/2WnfiJbxBw4aZkQ==
Sec-Fetch-Dest: websocket
Sec-Fetch-Mode: websocket
Sec-Fetch-Site: cross-site

The response, if accepted, is
HTTP/1.1 101 Switching Protocols
Upgrade: websocket
Connection: Upgrade
...

After that, the connection is kept open, and is now full duplex.
Since the handshake is a regular HTTP request, it is easy to integrate with any previously existing HTTP authentication scheme, in our case: add jwt authentication to the request. One thing I'm not sure about, though, is exactly what to sign. I guess nothing (empty payload), since the initial GET request does not have a json payload.

After having investigated/tested this a bit, I've come across a couple of problems.
From https://testdriven.io/courses/taxi-react/websockets-part-one/#H-3-authenticating-socket-connections :

Anything that can be sent with an HTTP request can be sent with the handshake -- i.e., headers and cookies, query string parameters, and request bodies. Unfortunately, the JavaScript WebSocket API does not support custom headers. That means we need to find a different way to authenticate our WebSocket connection than an authorization header.

https://developer.mozilla.org/en-US/docs/Web/API/WebSocket

So. the Authorization: Bearer <token> technically does work, but only with custom clients and/or curl examples, not with browser communication. I'm not sure if this is a blocker -- do we want to expose the engine api for browser interaction?
The suggested solution is to pass the token in url query parameters. This, however, means that the token will/can be exposed in proxy caches and/or server logs -- which further means that we can definitely not use the empty payload for initial connect, but rather something based on a nonce or a timestap.
I'm not sure what the best solution is here.
cc @fjl

I'm not sure if this is a blocker -- do we want to expose the engine api for browser interaction?

Engine API is planned to be used by CL client to interact with EL one. I don't see the case when browser plays a role of CL client

Regarding "key distribution" -- is the common flow to start one side (either EL or CL), grab the key, then start the other side with that key?
And advanced usage is to generate in some independent flow and provide the key on both sides?

Engine API is planned to be used by CL client to interact with EL one. I don't see the case when browser plays a role of CL client

There is a desire to see CL light clients in the browser but (1) I'm not sure if/when this will happen and (2) it's unclear if it would drive an external EL client in such a case

Regarding "key distribution" -- is the common flow to start one side (either EL or CL), grab the key, then start the other side with that key?
And advanced usage is to generate in some independent flow and provide the key on both sides?

That was my idea, yes.
The main point I'm not happy about right now is the empty payload at ws connect.

Perhaps it would be good to also include the iat claim in the JWT: https://datatracker.ietf.org/doc/html/rfc7519#section-4.1.6 . This means that the websocket connect token will not be identical each time, and the EL can verify the freshness of each token, and only allow access if the issuance time is within a certain time window, maybe 5 seconds in either direction.

Perhaps it would be good to also include the iat claim in the JWT

This seems like a reasonable approach to me

Is there any limits to what is a valid jwt-secret, or is any 256-bit value acceptable?

Is there any expectation to include exp (expiration) on the tokens?  Feels like we should, and it would be good to have some "general advice" for what a reasonable expiration time is.

Is there any limits to what is a valid jwt-secret, or is any 256-bit value acceptable?

Well, we can prohibit 0x00, but not sure what else we can do.

Is there any expectation to include exp (expiration) on the tokens

The idea with iat would be to make it so that there's a 5 (?) - second window where it is valid. So it would implicitly carry the exp. Or rather, the expiry would be enforced on the EL, not settable by the CL.
The canonical usecase for exp is when a federated server issues "this is John, he's an admin, token is valid for one hour". It's not quite our usecase

Is there anything else here to discuss?
We're looking to get a release out early this week to keep engineering moving. Shall we write up a PR

Yup. I can get started tomorrow at the earliest, if anyone wants to get it going before that please go ahead

I have concerns about JWT, it seems like a complex protocol with many options and so pitfalls.
For instance:

https://news.ycombinator.com/item?id=14292223
https://news.ycombinator.com/item?id=14291059
https://paragonie.com/blog/2017/03/jwt-json-web-tokens-is-bad-standard-that-everyone-should-avoid
https://redis.com/blog/json-web-tokens-jwt-are-dangerous-for-user-sessions/
https://github.com/dwyl/learn-json-web-tokens/blob/master/README.md
http://cryto.net/~joepie91/blog/2016/06/13/stop-using-jwt-for-sessions/
http://cryto.net/~joepie91/blog/2016/06/19/stop-using-jwt-for-sessions-part-2-why-your-solution-doesnt-work/
https://latacora.micro.blog/2018/06/12/a-childs-garden.html

Excerpts:

So, as someone who does some work in crypto engineering, arguments about JWT being problematic only if implementations are "bungled" or developers are "incompetent" are sort of an obvious "tell" that the people behind those arguments aren't really crypto people. In crypto, this debate is over.
I know a lot of crypto people who do not like JWT. I don't know one who does. Here are some general JWT concerns:

It's kitchen-sink complicated and designed without a single clear use case. The track record of cryptosystems with this property is very poor. Resilient cryptosystems tend to be simple and optimized for a specific use case.

It's designed by a committee and, as far as anyone I know can tell, that committee doesn't include any serious cryptographers. I joked about this on Twitter after the last JWT disaster, saying that JWT's support for static-ephemeral P-curve ECDH was the cryptographic engineering equivalent of a "kick me" sign on the standard. You could look at JWT, see that it supported both RSA and P-curve ECDH, and immediately conclude that crypto experts hadn't had a guiding hand in the standard.

Flaws in crypto protocols aren't exclusive to, but tend to occur mostly in, the joinery of the protocol. So crypto protocol designers are moving away from algorithm and "cipher suite" negotiation towards other mechanisms. Trevor Perrin's Noise framework is a great example: rather than negotiating, it defines a family of protocols and applications can adopt one or the other without committing themselves to supporting different ones dynamically. Not only does JWT do a form of negotiation, but it actually allows implementations to negotiate NO cryptography. That's a disqualifying own-goal.

JWT's defaults are incoherent. For instance: non-replayability, one of the most basic questions to answer about a cryptographic token, is optional. Someone downthread made a weird comparison between JWT and Nacl (weird because Nacl is a library of primitives, not a protocol) based on forward-security. But for a token, replayability is a much more urgent concern.

The protocol mixes metadata and application data in two different bag-of-attributes structures and generally does its best to maximize all the concerns you'd have doing cryptography with a format as malleable as JSON. Seemingly the only reason it does that is because it's "layered" on JOSE, leaving the impression that making a pretty lego diagram is more important to its designers than coming up with a simple, secure standard.

It's 2017 and the standard still includes X.509, via JWK, which also includes indirected key lookups.

The standard supports, and some implementations even default to, compressed plaintext. It feels like 2012 never happened for this project.

For almost every use I've seen in the real world, JWT is drastic overkill; often it's just an gussied-up means of expressing a trivial bearer token, the kind that could be expressed securely with virtually no risk of implementation flaws simply by hexifying 20 bytes of urandom. For the rare instances that actually benefit from public key cryptography, JWT makes a hard task even harder. I don't believe anyone is ever better off using JWT. Avoid it.

And another one

The problems, for people who don't want to read articles from comment links, are:

JSON Web Signing

alg headers
"This Header Parameter MUST be present and MUST be understood and processed by implementations."

JSON Web Encryption

RSA with PKCS1v1.5 padding (power word: Bleichenbacher 1998)
ECDH over NIST curves (and, in practice, invalid-curve attacks)
AES-GCM included in a list of asymmetric algorithm choices, for added confusion
AES-GCM for shared-key encryption, without guidance over nonces or key rotation

A better solution from JOSE would only give developers two options:

Version (v1, v2, v3, etc. which hard-coded the algorithm choices)
Operation

enc -> crypto_secretbox()
auth -> crypto_auth()
pub-enc -> crypto_box_seal()
pub-sign -> crypto_sign_detached()

@mratsim Do you have a recommendation for a good alternative solution?  One of the things I like about JWT is there is a library for every language (often multiple libraries).  While I agree with some of the arguments made in the links you provided about JWTs being overkill (especially since we aren't leveraging capabilities), I would like to see a concrete proposal for an alternative that is either so trivially simple that a library isn't necessary, or has a large selection of libraries that could be used.

TOTP?

AFAICS, this articles are explaining why JWT is bad for user authentication and in-browser use cases. I haven't found any issues in application to our particular use case, please, correct me if it's been overlooked. Our use case seems very simple and I don't think that if we were designing auth by ourselves without relying on already existing schemas we wouldn't end up with a pretty similar scheme that we have with JWT utilization.

The argument is basically that JWT is extremely unnecessary for our needs.  We literally just need a preshared key and a signature over the payload (or similar).  JWT provides things like capabilities, permissions, etc., All features we aren't going to use.

I think a very reasonable question to ask would be, "what features exactly do we need from the authentication?". I haven't seen that listed anywhere, is it?

@mkalinin

AFAICS, this articles are explaining why JWT is bad for user authentication and in-browser use cases. I haven't found any issues in application to our particular use case, please, correct me if it's been overlooked. Our use case seems very simple and I don't think that if we were designing auth by ourselves without relying on already existing schemas we wouldn't end up with a pretty similar scheme that we have with JWT utilization.

That's exactly why I think JWT is overkill and exposing us to many implementation bugs in third-party libraries and/or supply chain attacks while our use-case and threat model is very simple.
@MicahZoltu

Do you have a recommendation for a good alternative solution? One of the things I like about JWT is there is a library for every language (often multiple libraries). While I agree with some of the arguments made in the links you provided about JWTs being overkill (especially since we aren't leveraging capabilities), I would like to see a concrete proposal for an alternative that is either so trivially simple that a library isn't necessary, or has a large selection of libraries that could be used.

I've been looking into very simple schemes that would be misuse-resistant and came across Branca: https://github.com/tuupola/branca-spec
It only needs XChaCha20-Poly1305, has test vectors, has 3 libraries in Go, 2 in Java, 1 Javascript, 1 Python and 1 Rust among other languages.
The Python impl by the spec for reference is just 50 lines of code, assuming XChaCha20-Poly1305 provided by a crypto lib.
Tagging @asanso since he is mentioned by name in JWT best practices at https://tools.ietf.org/id/draft-ietf-oauth-jwt-bcp-02.html#insecure-use-of-elliptic-curve-encryption
https://github.com/tuupola/pybranca/blob/79366150353727c0410e111eef7dbaf2ed97ec8e/branca.py
"""
Branca
Authenticated and encrypted API tokens using modern crypto.
"""

import base62
import calendar
import ctypes
import struct
from binascii import unhexlify
from datetime import datetime
from xchacha20poly1305 import generate_nonce
from xchacha20poly1305 import crypto_aead_xchacha20poly1305_ietf_encrypt
from xchacha20poly1305 import crypto_aead_xchacha20poly1305_ietf_decrypt
from xchacha20poly1305 import CRYPTO_AEAD_XHCACHA20POLY1305_IETF_NPUBBYTES
from xchacha20poly1305 import CRYPTO_AEAD_XHCACHA20POLY1305_IETF_KEYBYTES

class Branca:
    VERSION = 0xBA

    def __init__(self, key):
        if isinstance(key, bytes):
            self._key = key
        else:
            self._key = unhexlify(key)

        if len(self._key) is not CRYPTO_AEAD_XHCACHA20POLY1305_IETF_KEYBYTES:
            raise ValueError(
                "Secrect key should be {} bytes long".format(
                    CRYPTO_AEAD_XHCACHA20POLY1305_IETF_KEYBYTES
                )
            )

        self._nonce = None # Used only for unit testing!

    def encode(self, payload, timestamp=None):

        if not isinstance(payload, bytes):
            payload = payload.encode()

        if timestamp is None:
            timestamp = calendar.timegm(datetime.utcnow().timetuple())

        version = struct.pack("B", self.VERSION)
        time = struct.pack(">L", timestamp)

        if self._nonce is None:
            nonce = generate_nonce()
        else:
            nonce = self._nonce

        header = version + time + nonce
        ciphertext = crypto_aead_xchacha20poly1305_ietf_encrypt(payload, header, nonce, self._key)

        return base62.encodebytes(header + ciphertext)

    def decode(self, token, ttl=None):
        token = base62.decodebytes(token)
        header = token[0:CRYPTO_AEAD_XHCACHA20POLY1305_IETF_NPUBBYTES + 5]
        nonce = header[5:CRYPTO_AEAD_XHCACHA20POLY1305_IETF_NPUBBYTES + 5]
        ciphertext = token[CRYPTO_AEAD_XHCACHA20POLY1305_IETF_NPUBBYTES + 5:]

        version, time = struct.unpack(">BL", bytes(header[0:5]))

        # Implementation should accept only current version.
        if version is not self.VERSION:
            raise RuntimeError("Invalid token version")

        payload = crypto_aead_xchacha20poly1305_ietf_decrypt(ciphertext, header, nonce, self._key)

        if ttl is not None:
            future = time + ttl
            timestamp = calendar.timegm(datetime.utcnow().timetuple())
            if future < timestamp:
                raise RuntimeError("Token is expired")

        return payload

    def timestamp(self, token):
        token = base62.decodebytes(token)
        version, time = struct.unpack(">BL", bytes(token[0:5]))

        return time

Note compared to the desired properties, Branca is also overkill in the following way:

Payload is encrypted. Depending on the volume of data between CL and EL this might be too much latency? But it might opens up new use-cases when CL and EL are on dedicated VM/machines and XChaCha20-Poly1305 is likely to become heavily optimized in the future as it's easier to make it fast compared to AES (which requires hardware acceleration to reach GB throughput https://www.bearssl.org/speed.html).

That's exactly why I think JWT is overkill and exposing us to many implementation bugs in third-party libraries and/or supply chain attacks while our use-case and threat model is very simple.

Vulnerabilities in a 3rd party libraries is a good point. The spec is currently based on HMAC + SHA256, vulnerabilities in JWT libraries are related to EC based algorithms. I am not against using custom auth scheme, using any other 3rd party solution is prone to vulnerabilities, though, may be less prone if the solution is tiny. Just want to be sure that JWT is not an option.

what features exactly do we need from the authentication?

Currently it is 1) be able to prove that CL knows a secret 2) basic replay protection for auth tokens (via "iat") 3) optional: versioning. I believe we don't need anything on top of that. cc @holiman

Btw, could we implement current auth scheme avoiding usage of 3rd party JWT libs? We have crypto libs and HMAC + SHA256 is widely supported, we have JSON libs and Base64 codecs -- all these dependencies should already be used by client implementations.

Tagging @asanso since he is mentioned by name in JWT best practices at https://tools.ietf.org/id/draft-ietf-oauth-jwt-bcp-02.html#insecure-use-of-elliptic-curve-encryption

Since I have been mentioned by @mratsim here my 2 cents.  I actually already gave a look to this proposal (thanks @djrtwo  for pointing this out) and while I share some of the concerns about JWT as a whole (as  you can also see from some of my previous research) the subset used in this proposal seem to be immune to those vulnerabilities.
So it might really just a quick win to just use JWT (specifically HMAC + SHA256) for this use case rather than invent yet another solution.

Btw, could we implement current auth scheme avoiding usage of 3rd party JWT libs? We have crypto libs and HMAC + SHA256 is widely supported, we have JSON libs and Base64 codecs -- all these dependencies should already be used by client implementations.

I think this is reasonable, though for censorship resistance and also to enable more distributed use-cases in the future I do think adding encryption would be interesting (and leaving key negotiation out of scope).

I do think adding encryption would be interesting (and leaving key negotiation out of scope)

We have been discussing this and ended up with TLS as a solution for encrypting communication channel between CL and EL. In this case extra auth scheme becomes redundant.

I do think adding encryption would be interesting (and leaving key negotiation out of scope)

We have been discussing this and ended up with TLS as a solution for encrypting communication channel between CL and EL. In this case extra auth scheme becomes redundant.

What about the certificates and the certificate authorities though? Or do we use self-signing certificates? In that case a pre-shared key is likely better because it allows end-users negotiating them with whatever they want, copy-pasting as suggested in this RFC or Diffie-Hellman in more complex infra.

What about the certificates and the certificate authorities though? Or do we use self-signing certificates? In that case a pre-shared key is likely better because it allows end-users negotiating them with whatever they want, cli-config/copy-pasting as suggested in this RFC or Diffie-Hellman in more complex infra.

I think there are options here and TLS with self-signed certificates is one of them. IMHO, more sophisticated encryption and auth schemas shouldn't be a part of client implementations and rather be set up as additional infra alongside with clients. For instance, it could be an HTTPS/WSS server in front of EL and a corresponding client on the other side that CL has access through

We discussed a few different schemes. One was to just go with TLS, but that would put tls endpoints into the client - making it difficult to integrate with a production TLS terminator, in case someone wants to put the api behind an nginx or cloudflare terminator. Not to mention certificate handling, which is a bit cumbersome.

Btw, could we implement current auth scheme avoiding usage of 3rd party JWT libs?

Definitely. This EIP mandates HMAC+SHA256, and iat property. It would be a pretty small thing to write a custom JWT handler which does only that. Whether client implementors choose to do that or go with a third party library is an implementation choice. The current go-ethereum PR uses a third party library, but explicitly only allows the HMAC+SHA256 scheme.
I've said from the start that I'm not hell-bent on JWT, but I still think it's suitable for this purpose -- given the constraints put forth in this EIP. There is ambiguity about cryptography schema used, and a 10-second validity window. It can easily be integrated with a regular off-the-shelf TLS-terminator/load balancer.
I'm still open to being convinced otherwise, if there are any concrete arguments about why our use of it is insufficient/insecure.

Although the concerns of JWT being a "kitchen-sink" approach that has valid criticisms, I do think the way we use it in the scope of the engine API authentication scheme is decent. It is extremely commonplace, has tons of library support, and it is also fairly simple to write a custom JWT handler that meets the needs of the spec here. The auth constraints are simple, and we are not building this for web. Upon first learning the spec was suggesting JWT, we were able to immediately add support for it in Prysm as almost every developer that has worked on http auth has heard of JWT. Branca, although simpler in scope, feels more niche.
TL;DR: based on the constraints of the spec, JWT works fine as there is no serious downside that necessitates using a more niche schema.

Btw -- I guess this ticket can be closed now, since the corresponding PR has been merged. Any further discussions on this should rather go to either a new issue or a PR replacing it with something else.


Change user repository pattern (use inheritance)

This is to illustrate the proposal of issue #5.
What do you think?
Cheers!
Sébastien.

I think it is a great idea / implementation.


Improving Power Automate tutorials and parameter guidance

This PR:

Fixes the Grade Calculator screenshots.
Gives more context to selecting a workbook location in the PA tutorials.
Gives examples for union behavior in the PA parameter section.

Docs Build status updates of commit dc0df18:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

docs/develop/power-automate-integration.md
:white_check_mark:Succeeded

docs/images/action-recorder-intro.png
:white_check_mark:Succeeded

docs/images/code-editor-intro.png
:white_check_mark:Succeeded

docs/images/scenario-grade-calculator-after.png
:white_check_mark:Succeeded

docs/resources/scenarios/grade-calculator.md
:white_check_mark:Succeeded

docs/tutorials/excel-power-automate-manual.md
:white_check_mark:Succeeded

docs/tutorials/excel-power-automate-trigger.md
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching in the Docs contributor and Admin GuideSee the frequently asked questionsPost your question in the Docs support channel


Revert "disable date test in ClusteredIndexTest (#1941)"

This reverts commit da92a61af2733fc4376e9295325e8289962a7d22.
What problem does this PR solve?
What is changed and how it works?
Check List
Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No code

Code changes

Has exported function/method change
Has exported variable/fields change
Has interface methods change
Has persistent data change

Side effects

Possible performance regression
Increased code complexity
Breaking backward compatibility

Related changes

Need to cherry-pick to the release branch
Need to update the documentation
Need to update the tidb-ansible repository
Need to be included in the release note

/run-all-tests


Service implementation joins not working

Can't figure out simple task: join 1 table and add column. There is no usefull documentation about service implementation.
Code snippet of problem
public function query()
    {
        $query = Technika::query()
            ->join('manufacturers','technika.manufacturer_id','=','manufacturers.id')
            ->select($this->getColumns());

        return $this->applyScopes($query);
    }

protected function getColumns()
    {
        return [
            'manufacturers.id',
        ];
    }

This will trigger bizarre error
Requested unknown parameter 'manufacturers.id' for row 0, column 0
Tried many variations like:
return [
    'id',
];

Above will trigger Column 'id' in field list is ambiguous
Another one was:
return [
  [
    'name' => 'id',
    'data' => 'id'
  ]
];

this will result in: strtolower() expects parameter 1 to be string, array given
And so on and so forth. Maybe some one jsut can give basic join example using Service implementation?
System details

Operating System OSX
PHP Version 7.2
Laravel Version 5.5
Laravel-Datatables Version 8.0

You need to specify the table name with column name when using join. See https://datatables.yajrabox.com/eloquent/joins for ref. The only diff on demo with service implementation is that the  scripts/columns are declared on the php side. Hope it makes sense. Thanks!

Thanks for follow up @yajra but this was first thing I've tried and when I set up columns like this:
protected function getColumns()
    {
        return [
            'technika.id',
            'manufacturers.title'
        ];
    }

Requested unknown parameter 'technika.id' for row 0, column 0
However response data is correct (as I can see in XHR response) but table renders empty rows.

Update
Solved problme by doing this:
protected function getColumns()
    {
        return [
            [ 'data' => 'id', 'name' => 'technika.id', 'title' => 'ID' ],
            [ 'data' => 'title', 'name' => 'manufacturers.title', 'title' => 'Manufacturer' ]
        ];
    }

public function query()
    {
        $query = Technika::query()
            ->join('manufacturers','technika.manufacturer_id','=','manufacturers.id')
            ->select( collect($this->getColumns())->pluck('name')->toArray() );

        return $this->applyScopes($query);
    }

getColumns method is used in query() and in html() and they both expect different type of array format.


Raspberry PI 3 support

I have followed steps depicted in https://github.com/flatcar-linux/Flatcar/issues/502 and in blog post mentioned on https://github.com/flatcar-linux/Flatcar/blob/main/community-meetings/2022-01-11-slides.pdf (BTW, the blog is offline, but the page is still available at https://github.com/littichokha/journalctl.dev/blob/main/content/posts/2021-12-25-flatcar-runs-on-raspberry-pi.md).
I've used RPI3 UEFI firmware from https://github.com/pftf/RPi3 instead of RPI4 firmware

Raspberry PI goes through UEFI, the GRUB, however it reboots after the following lines displayed on connected monitor:
Booting "Flatcar default'
EFI stub: Booting Linux Kernel...
EFI stub: Using DIB from configuration table
EFI stub: Exiting boot services and installing virtual address map...

As I understand, RPI3 also uses arm64 architecture, as it's compatible with Raspberry PI OS 64-bit.
Is it possible to add support for RPI v3, as there are still plenty of them around.
Thanks in advance!

I'm not super familiar with the state of UEFI and mainline kernels on rpi3, but the most likely culprit is the acpi=force flag passed to the kernel. Rpi3 might work in device tree mode. When in grub, try editing the menuentry and adding a line like this before linux:
set linux_cmdline="rootflags=rw mount.usrflags=ro consoleblank=0 $linux_root $linux_console $first_boot $randomize_disk_guid $oem $linux_append"

I removed extra_options from the default one, which contains the acpi=force arg.

@jepio it didn't help unfortunately. I've edited Flatcar default like this and pressed F2 to boot. It reboots just as previously in ~5 sec after EFI stub: Exiting boot services and installing virtual address map...

I guess the problem could be missing kernel config settings


feature/update-readme

Coverage remained the same at 89.381% when pulling 5df91274ccb9ed678e3a536212c95617f5475e49 on feature/update-readme into 36bf05bfee89484aae9b08d3ecf90d6e29b664d4 on master.

Codecov Report

Merging #9 into master will not change coverage.
The diff coverage is n/a.

@@          Coverage Diff           @@
##           master      #9   +/-   ##
======================================
  Coverage    85.5%   85.5%
======================================
  Files           4       4
  Lines          69      69
  Branches       20      20
======================================
  Hits           59      59
  Partials       10      10

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 36bf05b...5df9127. Read the comment docs.


Config File for DINO2reg-ViT-giant2

Can you please provide the config file for the DINO2reg-ViT-giant2. Currently there are config files only for: DINO2reg-ViT-Large and DINO2reg-ViT-Small.

Can you please provide the config file for the DINO2reg-ViT-giant2. Currently there are config files only for: DINO2reg-ViT-Large and DINO2reg-ViT-Small.

Updated now and please try it.

I am getting the following error:
Failed to find function: mono.model.backbones.vit_giant2_reg
when running: torch.nn.DataParallel(get_configured_monodepth_model(cfg, ))

I am getting the following error:
Failed to find function: mono.model.backbones.vit_giant2_reg
when running: torch.nn.DataParallel(get_configured_monodepth_model(cfg, ))

That's my mistake, I have added the model definitions now.


Refreshing a token does not work based on the proposed configuration

I ran into an issue with the proposed configuration. This is the suggested config:
OAUTH_AUTHORITY=https://login.microsoftonline.com/common
OAUTH_ID_METADATA=/v2.0/.well-known/openid-configuration
OAUTH_AUTHORIZE_ENDPOINT=/oauth2/v2.0/authorize
OAUTH_TOKEN_ENDPOINT=/oauth2/v2.0/token

wreck inside simple-oauth2 uses Url.URL to combine OAUTH_AUTHORITY with OAUTH_TOKEN_ENDPOINT which results in https://login.microsoftonline.com/oauth2/v2.0/token and therefore loses common. This results in a 404.
I changed the config slightly and removed the leading slashes from the relative paths and added a trailing slash to the base URL.
OAUTH_AUTHORITY=https://login.microsoftonline.com/common/
OAUTH_ID_METADATA=/v2.0/.well-known/openid-configuration
OAUTH_AUTHORIZE_ENDPOINT=oauth2/v2.0/authorize
OAUTH_TOKEN_ENDPOINT=oauth2/v2.0/token

So that OAUTH_TOKEN_ENDPOINT is relative. I have not figured why it worked for authorize though, but still works.

Interesting. I haven't run into this. What version of simple-oauth2 and wreck are you using?

It's "simple-oauth2": 3.1.0" and "hapi/wreck": 15.1.0

Hmm. Same here. What OS are you running on?

My devbox is windows 10. No production yet.

I was finally able to reproduce this. The reason authorize works is because the app uses passport for that, not simple-oauth2.
Thanks for reporting, I'll get a fix for this shortly.


Thanks

This is probably not the best way of doing this but I don't know how in GitHub to say thanks except to log an issue saying thanks for putting this WORKING example together. So much code out there on the web and this one actually works out of the box!

@marklapasa you're welcome :)


Add the ability to adjust bgRepeatInterval on Android

Calling Init() does PlatformScheduleFetch(), which schedules a periodic work request on Android. Currently, the repeat interval is 6 hours (bgRepeatInterval). We would like to change the repeat interval. There is a function called ConfigureBackgroundWorkRequest(), but it is not referenced anywhere and it's not accessible to the app using the cross-platform library. Is the existence of this function a sign that the ability to change bgRequestBuilder and bgRepeatInterval will be added soon?

You will need to access this from the Android app head project.  I'd put it in your MainActivity.cs file, something like this:
Xamarin.ExposureNotifications.ExposureNotification.ConfigureBackgroundWorkRequest(TimeSpan.FromHours(12), b =>
{
	b.SetPeriodStartTime(TimeSpan.FromSeconds(1))
	.SetConstraints(new AndroidX.Work.Constraints.Builder()
		.SetRequiredNetworkType(AndroidX.Work.NetworkType.Connected)
		.SetRequiresCharging(true)
		.Build());
});

This will cause the builder you set, to be used instead of the default one:

https://github.com/xamarin/xamarin.exposurenotification/blob/master/Xamarin.ExposureNotification/ExposureNotification.android.cs#L125-L126
https://github.com/xamarin/xamarin.exposurenotification/blob/master/Xamarin.ExposureNotification/ExposureNotification.android.cs#L144-L146

When the fetch is scheduled during the Init call: https://github.com/xamarin/xamarin.exposurenotification/blob/master/Xamarin.ExposureNotification/ExposureNotification.shared.cs#L60

@havart-netcompany can you confirm this provides you with what you need?

@Redth I can confirm that your suggestion provides me with what I need


How the effect calls another effect and gets the return value?

Forgive me for not finding reliable information on the network.

You can get the return-value simply as follows:
function* foo() {
  return 'return-value-of-foo'
}

function* bar() {
  const value = yield call(foo)
  console.assert(value === 'return-value-of-foo')
}

@shinima Tks.
Can the put method do the same thing?

Yielding a put effect will returns the return-value of store.dispatch(action). In most situations, store.dispatch simply returns the action back and you don't need to care it.
There are some edge cases that you are using redux-saga with other redux middlewares, and the return-value of store.dispatch may be changed by other middlewares.

For some reason, I can only use put to call other effects. Is the Promise it returns reliable?

If some other redux middleware returns a Promise, you can use put.resolve to wait for its resolution.
Say we have a reduxFecthMiddleware that returns a Promise for FETCH as follows:
const reduxFecthMiddleware = ({ dispatch }) => next => action => {
  if (action.type === "FETCH") {
    return fetch(action.url);
  }
  return next(action);
};

function* gen() {
  try {
    // we can use `put.resolve` to wait for the Promise's resolution
    const response = yield put.resolve({ type: "FETCH", url: "bilibili.com" });
  } catch (error) {
    // handle error
  }
}

In my opinion, put.resolve is somewhat complicated and we can avoid using it if we change our middleware from returning a Promise to dispatching another action when some asynchronous task is done.

I fully understand what I want to know now, thank you so much.


Update chart to reference DEIS_DEBUG env var rename

Renamed the debug env var.
workflow-dev should have default DEIS_DEBUG=true
workflow-betaX should have default DEIS_DEBUG=false
Refs https://github.com/deis/builder/pull/317
Refs https://github.com/deis/slugbuilder/pull/70

moving to rc1 with the other PRs that work on this


Java 11 support

Able to compile in Java 11 after added jaxb libraries, but facing issues in test cases due to JMockit.
JMockit Startup class has configuration till Java 8, above any version it will be considering as below JDK 6. Please refer the screen shot.

May be required some good amount of changes to migrate from com.googlecode.jmockit library to org.jmockit.
Able to validate the build locally by skipping the test cases mvn clean package -DskipTests.

Thanks for this contribution. The problem with the compatibility of the test framework is why I have not updated this repo to Java11. I'm not willing to merge your changes, if it means we cannot run tests.


Using a similar setup for UDM Pro VPN?

Thank you for the useful script and instructions. I'm wondering if we could use a similar approach for UDM Pro's VPN? My goal is to use the Cloudflare proxied DNS, instead of a public address.

Any dynamic DNS client that supports dyndns2, which should be virtually all, should work. It probably uses ddclient.

I think this a feature in the upcoming update to UniFi OS (see info on Teleport).
I always defer to feature updates from the manufacturer - they're the ones actually getting paid for this stuff 😜


Improve and expand Neovim documentation

An alternative to https://github.com/denoland/manual/pull/57#issuecomment-935538472 as requested by @bartlomieju
This update contains important information about Neovim 0.6 nightly and explains how to enable linting. It hopefully makes everything easier to understand by introducing a separate heading.
Also, the phrase Outside of is a redundancy I could not help but correct :)

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


top_k_docs=-1 leads to CUDA Out of Memory

Hello maintainers,
I have encountered an issue when trying to prompt the Llama2 model. While I can successfully prompt the model after uploading a single document, I run into a CUDA out of memory error when I attempt to do the same after uploading two documents.  Tried to allocate 1.27 GiB (GPU 0; 19.50 GiB total capacity; 16.04 GiB already allocated; 277.88 MiB free; 18.15 GiB reserved in total by PyTorch)
If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

Notably, the memory usage seems excessively high considering I'm only processing two documents. Additionally, it's worth mentioning that:

Both documents are identical
The documents are in PDF format
They are scans of rare documents and have not been subjected to OCR beforehand

View Stack Trace
load INSTRUCTOR_Transformer
max_seq_length  512
Xformers is not installed correctly. If you want to use memory_efficient_attention to accelerate training use the following command to install Xformers
pip install xformers.
The model 'OptimizedModule' is not supported for .
Number of requested results 1000 is greater than number of elements in index 16, updating n_results = 16
thread exception: (<class 'torch.cuda.OutOfMemoryError'>, OutOfMemoryError('CUDA out of memory. Tried to allocate 1.06 GiB (GPU 0; 19.50 GiB total capacity; 15.28 GiB already allocated; 1.01 GiB free; 17.41 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF'), <traceback object at 0x7fa4ee722b00>)
make stop: (<class 'torch.cuda.OutOfMemoryError'>, OutOfMemoryError('CUDA out of memory. Tried to allocate 1.06 GiB (GPU 0; 19.50 GiB total capacity; 15.28 GiB already allocated; 1.01 GiB free; 17.41 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF'), <traceback object at 0x7fa4ee722b00>)
hit stop
Traceback (most recent call last):
  File "/h2ogpt_conda/lib/python3.10/site-packages/gradio/routes.py", line 488, in run_predict
    output = await app.get_blocks().process_api(
  File "/h2ogpt_conda/lib/python3.10/site-packages/gradio/blocks.py", line 1431, in process_api
    result = await self.call_function(
  File "/h2ogpt_conda/lib/python3.10/site-packages/gradio/blocks.py", line 1123, in call_function
    prediction = await utils.async_iteration(iterator)
  File "/h2ogpt_conda/lib/python3.10/site-packages/gradio/utils.py", line 349, in async_iteration
    return await iterator.__anext__()
  File "/h2ogpt_conda/lib/python3.10/site-packages/gradio/utils.py", line 342, in __anext__
    return await anyio.to_thread.run_sync(
  File "/h2ogpt_conda/lib/python3.10/site-packages/anyio/to_thread.py", line 33, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "/h2ogpt_conda/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 877, in run_sync_in_worker_thread
    return await future
  File "/h2ogpt_conda/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 807, in run
    result = context.run(func, *args)
  File "/h2ogpt_conda/lib/python3.10/site-packages/gradio/utils.py", line 325, in run_sync_iterator_async
    return next(iterator)
  File "/h2ogpt_conda/lib/python3.10/site-packages/gradio/utils.py", line 694, in gen_wrapper
    yield from f(*args, **kwargs)
  File "/workspace/src/gradio_runner.py", line 2323, in bot
    for res in get_response(fun1, history):
  File "/workspace/src/gradio_runner.py", line 2272, in get_response
    for output_fun in fun1():
  File "/workspace/src/gen.py", line 1903, in evaluate
    for r in run_qa_db(
  File "/workspace/src/gpt_langchain.py", line 2625, in _run_qa_db
    raise thread.exc
  File "/workspace/src/utils.py", line 398, in run
    self._return = self._target(*self._args, **self._kwargs)
  File "/h2ogpt_conda/lib/python3.10/site-packages/langchain/chains/base.py", line 258, in __call__
    raise e
  File "/h2ogpt_conda/lib/python3.10/site-packages/langchain/chains/base.py", line 252, in __call__
    self._call(inputs, run_manager=run_manager)
  File "/h2ogpt_conda/lib/python3.10/site-packages/langchain/chains/combine_documents/base.py", line 106, in _call
    output, extra_return_dict = self.combine_docs(
  File "/h2ogpt_conda/lib/python3.10/site-packages/langchain/chains/combine_documents/stuff.py", line 165, in combine_docs
    return self.llm_chain.predict(callbacks=callbacks, **inputs), {}
  File "/h2ogpt_conda/lib/python3.10/site-packages/langchain/chains/llm.py", line 252, in predict
    return self(kwargs, callbacks=callbacks)[self.output_key]
  File "/h2ogpt_conda/lib/python3.10/site-packages/langchain/chains/base.py", line 258, in __call__
    raise e
  File "/h2ogpt_conda/lib/python3.10/site-packages/langchain/chains/base.py", line 252, in __call__
    self._call(inputs, run_manager=run_manager)
  File "/h2ogpt_conda/lib/python3.10/site-packages/langchain/chains/llm.py", line 92, in _call
    response = self.generate([inputs], run_manager=run_manager)
  File "/h2ogpt_conda/lib/python3.10/site-packages/langchain/chains/llm.py", line 102, in generate
    return self.llm.generate_prompt(
  File "/h2ogpt_conda/lib/python3.10/site-packages/langchain/llms/base.py", line 451, in generate_prompt
    return self.generate(prompt_strings, stop=stop, callbacks=callbacks, **kwargs)
  File "/h2ogpt_conda/lib/python3.10/site-packages/langchain/llms/base.py", line 582, in generate
    output = self._generate_helper(
  File "/h2ogpt_conda/lib/python3.10/site-packages/langchain/llms/base.py", line 488, in _generate_helper
    raise e
  File "/h2ogpt_conda/lib/python3.10/site-packages/langchain/llms/base.py", line 475, in _generate_helper
    self._generate(
  File "/h2ogpt_conda/lib/python3.10/site-packages/langchain/llms/base.py", line 961, in _generate
    self._call(prompt, stop=stop, run_manager=run_manager, **kwargs)
  File "/h2ogpt_conda/lib/python3.10/site-packages/langchain/llms/huggingface_pipeline.py", line 168, in _call
    response = self.pipeline(prompt)
  File "/h2ogpt_conda/lib/python3.10/site-packages/transformers/pipelines/text_generation.py", line 201, in __call__
    return super().__call__(text_inputs, **kwargs)
  File "/h2ogpt_conda/lib/python3.10/site-packages/transformers/pipelines/base.py", line 1120, in __call__
    return self.run_single(inputs, preprocess_params, forward_params, postprocess_params)
  File "/h2ogpt_conda/lib/python3.10/site-packages/transformers/pipelines/base.py", line 1127, in run_single
    model_outputs = self.forward(model_inputs, **forward_params)
  File "/h2ogpt_conda/lib/python3.10/site-packages/transformers/pipelines/base.py", line 1026, in forward
    model_outputs = self._forward(model_inputs, **forward_params)
  File "/workspace/src/h2oai_pipeline.py", line 159, in _forward
    return self.__forward(model_inputs, **generate_kwargs)
  File "/workspace/src/h2oai_pipeline.py", line 195, in __forward
    generated_sequence = self.model.generate(input_ids=input_ids, attention_mask=attention_mask, **generate_kwargs)
  File "/h2ogpt_conda/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/h2ogpt_conda/lib/python3.10/site-packages/transformers/generation/utils.py", line 1522, in generate
    return self.greedy_search(
  File "/h2ogpt_conda/lib/python3.10/site-packages/transformers/generation/utils.py", line 2339, in greedy_search
    outputs = self(
  File "/h2ogpt_conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/h2ogpt_conda/lib/python3.10/site-packages/accelerate/hooks.py", line 165, in new_forward
    output = old_forward(*args, **kwargs)
  File "/h2ogpt_conda/lib/python3.10/site-packages/transformers/models/llama/modeling_llama.py", line 688, in forward
    outputs = self.model(
  File "/h2ogpt_conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/h2ogpt_conda/lib/python3.10/site-packages/accelerate/hooks.py", line 165, in new_forward
    output = old_forward(*args, **kwargs)
  File "/h2ogpt_conda/lib/python3.10/site-packages/transformers/models/llama/modeling_llama.py", line 578, in forward
    layer_outputs = decoder_layer(
  File "/h2ogpt_conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/h2ogpt_conda/lib/python3.10/site-packages/accelerate/hooks.py", line 165, in new_forward
    output = old_forward(*args, **kwargs)
  File "/h2ogpt_conda/lib/python3.10/site-packages/transformers/models/llama/modeling_llama.py", line 292, in forward
    hidden_states, self_attn_weights, present_key_value = self.self_attn(
  File "/h2ogpt_conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/h2ogpt_conda/lib/python3.10/site-packages/accelerate/hooks.py", line 165, in new_forward
    output = old_forward(*args, **kwargs)
  File "/h2ogpt_conda/lib/python3.10/site-packages/transformers/models/llama/modeling_llama.py", line 231, in forward
    attn_weights = nn.functional.softmax(attn_weights, dim=-1, dtype=torch.float32).to(query_states.dtype)
  File "/h2ogpt_conda/lib/python3.10/site-packages/torch/nn/functional.py", line 1845, in softmax
    ret = input.softmax(dim, dtype=dtype)
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 1.06 GiB (GPU 0; 19.50 GiB total capacity; 15.28 GiB already allocated; 1.01 GiB free; 17.41 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

Using --top_k_docs=-1 is filling up the context of the model, so uses maximum amount of memory.  You have a few options:

Leave top_k_docs as default of 3 to 4, or set it
Use smaller model
Use same model but quantized like AutoGPTQ from TheBloke using exllama or GGML from TheBloke
Use 4-bit or 8-bit, esp. 4-bit is quite efficient and after recent fixes is quite fast.

Thank you for the clarification on the --top_k_docs=-1 issue. I appreciate the suggested solutions.
However, I'd like to point out some ambiguities in the naming and definitions provided:

:param top_k_docs: number of chunks to give LLM
:param chunk: Whether to chunk data (True unless know data is already optimally chunked)
:param chunk_size: Size of chunks, with typically top-4 passed to LLM, so needs to be in context length

From the above definition, it appears that top_k_docs might be more appropriately named top_k_chunks.
Considering a large document could be divided into hundreds of chunks based on the chunk and chunk_size parameters, it's essential to have a clear distinction.
Furthermore, it seems intuitive that there should be a way to process the chunks serially, especially if the model's context is being filled up. This would allow for processing without overloading the memory.

top_k_docs started as number of chunks, but it applies to summarization too now, when it is not chunks but pages or other document parts however the parser did its job.  So it's not always chunk number.  I can update doc string.
Managing parameters to fit into memory is generally extraordinarily challenging.  However,  you can vary max_seq_len to limit your use of the model's sequence length while still using top_k_docs=-1 to auto-fill that context.


build(deps): bump controller-tools version to 0.14.0

Welcome to Cryostat! 👋
Before contributing, make sure you have:

[x] Read the contributing guidelines
[x] Linked a relevant issue which this PR resolves
[x] Linked any other relevant issues, PR's, or documentation, if any
[x] Resolved all conflicts, if any
[x] Rebased your branch PR on top of the latest upstream main branch
[x] Attached at least one of the following labels to the PR: [chore, ci, docs, feat, fix, test]
[x] Signed all commits: git commit -S -m "YOUR_COMMIT_MESSAGE"

Related to #849
Description of the change:

Bumped controller-tools to 0.14.0 to be compatible with Go 1.22+

/build_test

/build_test

https://github.com/cryostatio/cryostat-operator/actions/runs/9406013939